Sum Rate Optimization for Multi-IRS-Aided Multi-BS Communication System Based on Multi-Agent

Abstract

Intelligent reflecting surface (IRS) is a revolutionizing technology for improving the spectral and energy efficiency of future wireless networks. In this paper, we consider a downlink large-scale system empowered by multi-IRS to aid communication between the multiple base stations (BSs) and multiple user equipment (UEs). We target maximizing the sum rate by jointly optimizing the UE association, the transmit powers of BSs, and the configurations of the IRS beamforming. Due to the applicability restrictions of conventional optimization methods and their high complexity with large-scale networks in dynamic environments, deep reinforcement (DRL) learning is adopted as an alternative approach to finding optimal solutions. First, we model the optimization problem as a multi-agent Markov decision problem (MAMDP). Then, because large-scale wireless networks are naturally complex and changeable, and because many entities interact and affect how the whole system works, it is important to use a multi-agent approach to understand the complex dependencies and relationships between the different parts. In order to solve the problem, we propose a cooperative multi-agent deep reinforcement learning (MADRL)-based algorithm that works for both continuous and discrete IRS phase shifts. Simulation results validate that the proposed algorithm surpasses iterative optimization benchmarks regarding both sum rate performance and convergence.

1. Introduction

In light of the escalating need for enhanced data rates, improved energy and spectrum efficiency, and pervasive network access in the upcoming network generations [1], researchers have suggested numerous enhanced solutions encompassing various technologies. Among these, the intelligent reflecting surface (IRS) has emerged as a highly appealing and attractive technology that reconfigures the wireless propagation environment via software-controlled reflections [2]. In particular, an IRS is made up of many passive reflecting elements, and each one can change the amplitude and phase angles of the signals it receives on its own [3]. The IRS can create an extra communication path between the BSs and the UEs, which keeps communication from getting stuck and improves signal reception at the intended UEs and attenuating it at the non-intended UEs [4]. Moreover, due to its inherent properties and notable benefits in the aspects of spectral and energy efficacy, along with its economically viable deployment, the IRS is anticipated to surpass alternative technologies like backscatters and relays [5], making it a superior technology.

Numerous recent works have employed traditional optimization techniques for the joint optimization design within wireless networks, aided by the IRS. Specifically, in the study [4], the semidefinite (SDR)-based relaxation and alternating techniques were utilized to address the BS and IRS beamforming. Expanding on this joint design concept, the authors in [6] introduced a difference-of-convex approach within an IRS-aided system, overcoming the SDR limitations and efficiently handling the problem. Similarly, the work [7] concurrently configured the power allocation and IRS phases, employing gradient descent, fractional programming, and alternating maximization-based methods. The study [8] delved into optimizing the system’s weighted sum rate, employing an alternating approach-based Lagrangian transform. Additionally, certain works explored systems assisted by multi-IRS. In the study [9], Riemannian manifold and Lagrangian techniques were employed to configure the IRS phase shifts and transmit powers. The above works focused on a single BS scenario and ignored the interference from other BSs.

For the joint optimization problem in the multi-BS scenario, the UE association is important; hence, some works investigated joint optimization by including association design. Specifically, the researchers in [10] jointly designed passive beamforming and UE association in IRS-aided heterogeneous networks, where fixed point iteration and priority-based swapping algorithms were proposed. In [11], the researchers analyzed the average SINR for multiple-UE in a distributed IRS-assisted system. They used the successive refinement algorithm to find the best IRS–UE association coefficients and the best IRS configuration to serve the UEs that were connected to them. The proposed strategy maintained the association between the IRS and the UE for long periods, which minimized performance degradation and channel estimation overhead. In [12], an alternating-based approach was employed to optimize both the UE association and the passive beamforming of the assisting IRS for rate maximization. The work in [13] addressed the joint association problem in a multi-IRS-aided downlink system, where the association design between the BSs, UEs, and IRSs was designed to maximize the UEs’ utility via both optimal and sub-optimal presented solutions. In [14], the authors addressed the IRS–UE association problem to find the best balance of passive beamforming gains between various BS–UE links. Furthermore, the authors derived an average SINR (ASAINR) expression for each UE to formulate the ASAINR balancing problem, solving the IRS–UE association and power control. Here, all involved channels were assumed to be Rayleigh-fading in the analysis. The work [15] investigated an IRS-assisted multi-BS multi-UE millimeter wave downlink communication system. The authors formulated a strategy to maximize the sum rate by concurrently optimizing association design, power allocation, and IRS passive beamforming. They introduced an iterative algorithm grounded in alternating optimization, sequential fractional programming, and forward-reverse auction mechanisms to tackle this formulated problem. The study conducted in [16] examined a mmWave cellular network assisted by an IRS. In this configuration, an IRS was deployed to improve the signals to the UEs within each cell. Moreover, the UE association was optimized by a matching game approach to balance the BS loads and increase the network utility.

Although the aforementioned conventional optimization-based methods in various approaches and algorithms have shown satisfactory performance, the system complexity often hinders their practical implementation [7]. Consequently, researchers have turned their focus towards machine learning (ML) techniques, which have proven to be effective at overcoming the challenges faced by traditional methods and reducing optimization complexity in implementing IRS-aided systems [2,17]. Among the ML approaches, deep reinforcement learning (DRL) is commonly applied to solve complex dynamic problems in IRS-aided systems due to its exceptional ability to learn the hidden patterns in the data. Within the works of optimizing IRS reflection phases, a study by the authors [18] introduced a DRL-based algorithm. This algorithm aimed to predict IRS reflection phases while minimizing the training overhead, in contrast to the supervised-based approach presented in [19]. In the study [20], an approach based on DRL was introduced for IRS reflection. This approach aimed for throughput maximization in an imperfect channel state information (CSI) scenario. In the context of the joint beamforming design, the study presented in [21] developed a DRL model. The overarching objectives were to enhance the system’s sum rate while maintaining a low level of complexity. The study outlined in [22] revolved around a secure wireless communication system incorporating an IRS. Enhancement of the system’s rate was achieved via the setup of both BS beamforming and IRS reflecting beamforming over various qualities of service (QoSs). The work in [23] formulated the energy efficiency maximization problem constrained to UEs’ data rates by jointly setting up the transmit powers, IRS configuration design, and deployment. To solve this joint problem, the authors adopted a long, short-term memory-based echo state network algorithm and a double-deep Q-network. Additionally, the authors of [24,25] optimized the designed ML-based algorithms for multi-IRS-aided communication system s. Specifically, the proposed GRNN-based and DDPG-based algorithms were presented for static and dynamic UE scenarios.

The above-mentioned works predominantly center around the single-agent paradigm, emphasizing the optimization and deployment of IRS-aided systems through individual agent strategies. However, when it comes to addressing challenges in IRS-aided systems, the single-agent paradigm encounters limitations in effectively navigating the complexities, grappling with intricate interdependencies, and adapting to the dynamic nature of wireless communication networks; hence, researchers are increasingly turning to the multi-agent framework. The shift to multi-agent systems becomes crucial in IRS-rich environments, where the collaborative coordination of multi-IRS holds the potential for significant improvements. Given the dynamic nature of wireless communication networks and the intricate interdependencies between IRSs and BSs, adopting multi-agent approaches offers distinct advantages. The deployment of a multi-agent approach facilitates collaborative decision-making, enabling a more holistic exploration of the solution space for refining the optimization of efficient IRS-aided communication systems. This, in turn, prompts some researchers to leverage the multi-agent paradigm [26,27,28,29,30]. Specifically, in [30], researchers presented a multi-agent-based deep Q-network approach that optimized the powers of the BSs, considering discrete power levels. There is a scarcity of literature dedicated to multi-agent algorithms in IRS-assisted systems [31,32]. In [31], the considered problem of power allocation and subcarrier assignment was formulated and solved by a multi-agent-based approach, while in [32], a multi-agent reinforcement learning-based buffer was proposed for aiding relay selection in an IRS-aided system. Precisely, the agents were trained to optimize the IRS reflection phases and relay selection to achieve maximum throughput.

While the existing literature has made significant strides in enhancing communication systems through the integration of IRS with ML, it is crucial to address certain limitations in the current research landscape. Many researchers have focused on optimizing IRS-aided communication in small-scale network scenarios. However, the practical deployment of communication systems often involves large-scale networks with multiple interacting elements. Existing studies have commonly built and trained their ML-based models based on fixed associations designed between systems nodes or discretizing their variables to defined levels. This approach might be inappropriate or not relevant for the deployed systems. The need for better spectral efficiency in communication networks is growing all the time. This makes studying the sum rate maximization problem even more important, since it directly relates to the main goal of improving system performance in large networks with multiple IRSs. Thus, in our work, we study the sum rate maximization problem for a large-scale system with multiple IRSs, where the phase of the IRS can be either discrete or continuous. A MADRL-based algorithm is proposed to solve the joint optimization of the transmit powers of the BSs, the reflection beamforming of the IRSs, and the design of BS–IRS–UE associations. The main contribution of this paper can be outlined as follows:

• We develop a large-scale communication system with multi-BS assisted by multi-IRS to serve multi-UE. We formulate a comprehensive optimization problem aimed at maximizing the data rates of served UEs. The joint optimization encompasses BSs’ transmit powers, IRSs’ passive beamforming, and the design of BS–UE–IRS associations, all subject to the system and UEs’ QoS requirements.
• Addressing the inherent non-convexity of the original optimization problem, we propose a MADRL-based algorithm under continuous and discrete phases of the IRS scenarios. The optimization problem is modeled as an MAMDF, where the agents are designed by integrating DDPG and DQN algorithms to learn a joint policy. This approach enables the system to dynamically adapt to changing environmental conditions and maximize the long-term total system utility.
• We analyze the convergence, computational complexity, and implementation efficiency of the proposed MADRL-based algorithm, integrated with the DRL algorithms. Our numerical results not only showcase the superior performance of our algorithm over benchmark schemes (e.g., the successive refinement-based approach), but also demonstrate its efficiency in terms of implementation runtime, thus validating its practical feasibility. Moreover, the achieved rate in the IRS-aided system exhibits substantial improvements as the number of reflecting elements increases, highlighting the scalability and effectiveness of the proposed approach.

The rest of this paper is structured as follows: Section 2 outlines the system model. Section 3 presents the formulated optimization problem. Section 4 details the proposed MADRL-based algorithm for continuous and discrete phases of the IRS configurations. Following that, Section 5 presents the simulation results. Finally, Section 6 provides a summary of the work.

Notation: Italicized letters represent scalars, while boldface is used for vectors and matrices. The ${\left(.\right)}^T$ and ${(.)}^H$ superscripts indicate transpose and Hermitian operations, respectively, while $\mathbb{C}$ signifies the complex domain. The notation $\mathcal{CN}\left(\mu ,{\sigma }^2\right)$ denotes a circularly symmetric complex Gaussian (CSCG) distribution with mean $\mu$ and variance ${\sigma }^2$. The notations $\parallel .\parallel$, $\left|.\right|$, $\lfloor .\rfloor$, and $\mathbb{E}[.]$ represent the Euclidean norm, absolute value, floor, and expectation, respectively. Moreover, Mod(.) represents the modulo operation; $\mathrm{diag}\left(\mathit{x}\right)$ denotes a diagonal matrix constructed with the elements of the vector $\mathit{x}$ as its diagonal entries; $log(.)$ denotes the logarithm function; $\nabla f$ represents the gradient of f; and the order of complexity is denoted by $\mathcal{O}(.)$.

2. System Model

As shown in Figure 1, we consider a downlink large-scale system assisted by multi-IRS to aid the communication between the multiple BSs and multiple UEs. Let the sets of multi-BS, multi-UE, and multi-IRS be denoted as $\mathcal{S}=\{1,\dots ,S\}$, $\mathcal{K}=\{1,\dots ,K\}$, and $\mathcal{L}=\{1,\dots ,L\}$, respectively. At each time instance, the set of UEs can be divided into $K_s$ serving UEs and $K_w$ waiting UEs, with sets ${\mathcal{K}}_s=\{1,\dots ,K_s\}$, and ${\mathcal{K}}_w=\{1,\dots ,K_w\}$, respectively, where ${\mathcal{K}}_s\cup {\mathcal{K}}_w=\mathcal{K}$. Each BS is equipped with M transmit antennas, and each UE is equipped with a single antenna. Each IRS is a 2D surface composed of $N=N_H{N}_V$ reflecting elements arranged in a rectangular grid. Here, $N_H$ represents the number of elements per row, and $N_V$ represents the number of elements per column. In addition, a wireless controller connects to the BS to configure the elements of the IRS [33]. Let the binary variables ${\rho }_{k,s},k\in \mathcal{K},s\in \mathcal{S}$, represent the association indicator between the k-th UE and the s-th BS, where ${\rho }_{k,s}=1$ indicates that the k-th UE is associated with the s-th BS; otherwise, ${\rho }_{k,s}=0$. It is assumed that each UE can be served from a maximum of one BS and, in turn, each BS is restricted to serving only one UE at any given time instant. The binary variables, ${\lambda }_{l,s},l\in \mathcal{L},s\in \mathcal{S}$, denote the association indicator between the l-th IRS and the s-th BS, where ${\lambda }_{l,s}=1$ indicates that the l-IRS is associated with the s-th BS; otherwise, ${\lambda }_{l,s}=0$. Furthermore, it is assumed that each IRS can be associated with only one BS at a given time instant. Accordingly, we can define the constraints of BS–UE–IRS associations at each node as follows:

$$\begin{array}{cccccc}\hfill & \sum _{s\in \mathcal{S}}{\rho }_{k,s}\le 1,\hfill & \hfill & \forall k\in \mathcal{K}\hfill & \hfill & (\mathrm{UE}\mathrm{association}),\hfill \\ \hfill & \sum _{k\in \mathcal{K}}{\rho }_{k,s}\le 1,\hfill & \hfill & \forall s\in \mathcal{S}\hfill & \hfill & (\mathrm{BS}\mathrm{association}),\hfill \\ \hfill & \sum _{s\in \mathcal{S}}{\lambda }_{l,s}=1,\hfill & \hfill & \forall l\in \mathcal{L}\hfill & \hfill & (\mathrm{IRS}\mathrm{association}).\hfill \end{array}$$

(1)

We assume quasi-static, flat-fading channels. Specifically, the direct channel is denoted as ${\mathit{h}}_{s,dk}^H\in {\mathbb{C}}^{1\times M}$, which represents the channel from the s-th BS to the k-th UE, while the indirect channels, ${\mathit{G}}_{s,l}\in {\mathbb{C}}^{N\times M}$ and ${\mathit{h}}_{rkl}^H\in {\mathbb{C}}^{1\times N}$, represent the channels from the s-th BS to the l-th IRS and from the l-th IRS to the k-th UE, respectively. As for the channel models, the BS–UE channels are modeled as Rayleigh fading, ${\mathit{h}}_{s,dk}^H={\mathrm{PL}}_{s,k}{\tilde{\mathit{h}}}_{s,dk}^H$, where ${\mathrm{PL}}_{s,k}$ is the s-th BS-k-th UE channel path loss, and ${\tilde{h}}_{s,dk}^H$ follows $\mathcal{CN}(0,1)$. The IRSs are positioned to maintain line-of-sight (LOS) connections between the BSs and the UEs. Consequently, the channels for BS–IRS and IRS–UE follow Rician fading, ${\mathit{G}}_{s,l}={\mathrm{PL}}_{s,l}(\sqrt{\frac{\epsilon }{1+\epsilon }}{\tilde{\mathit{G}}}_{s,l}^{\left(\mathrm{LOS}\right)}+\sqrt{\frac{1}{1+\epsilon }}{\tilde{\mathit{G}}}_{s,l}^{\left(\mathrm{NLOS}\right)})$ and ${\mathit{h}}_{rkl}^H={\mathrm{PL}}_{k,l}(\sqrt{\frac{\epsilon }{1+\epsilon }}{\tilde{\mathit{h}}}_{rkl}^{H\left(\mathrm{LOS}\right)}+\sqrt{\frac{1}{1+\epsilon }}{\tilde{\mathit{h}}}_{rkl}^{H\left(\mathrm{NLOS}\right)})$, respectively. Here, ${\mathrm{PL}}_{s,l}$ and ${\mathrm{PL}}_{k,l}$ are the path losses of the s-th BS to the l-th IRS channel and the l-th IRS to the k-th UE channel, respectively, and $\epsilon$ is the Rician factor. The NLOS components, ${\tilde{\mathit{G}}}_{s,l}^{\mathrm{NLOS}}$ and ${\tilde{\mathit{h}}}_{rkl}^{H\left(\mathrm{NLOS}\right)}$, exhibit a distribution following $\mathcal{CN}(0,1)$. The LOS elements are formulated as per [34], that is, ${\tilde{\mathit{G}}}_{s,l}^{\mathrm{LOS}}={\mathit{a}}_{\mathrm{IRS}}({\vartheta }_{s,l},{\psi }_{s,l}){\mathit{a}}_{\mathrm{BS}}{({\vartheta }_s,{\psi }_s)}^H$ and ${\tilde{\mathit{h}}}_{rkl}^{H\left(\mathrm{LOS}\right)}={\mathit{a}}_{\mathrm{IRS}}{({\vartheta }_{l,k},{\psi }_{l,k})}^H$. Here, ${\mathit{a}}_{\mathrm{IRS}}$ and ${\mathit{a}}_{\mathrm{BS}}$ are the steering vectors for IRS and BS, respectively. The symbols $\vartheta$ and $\psi$ represent the azimuth and elevation angles. For the l-th IRS steering vector, the n-th element can be calculated as ${\left[{\mathit{a}}_{\mathrm{IRS}}({\vartheta }_{l,k},{\psi }_{l,k})\right]}_n=e^{j\frac{2\pi d^l}{{\lambda }_c}\{i_1\left(n\right)sin\left(\vartheta l,k\right)cos\left(\psi l,k\right)+i_2\left(n\right)sin\left({\psi }_{l,k}\right)\}}$, where ${\lambda }_c$ denotes the carrier wavelength, $d^l$ represents the distance between adjacent elements of the l-th IRS, $i_1\left(n\right)=mod(n-1,N_H)$, and $i_2\left(n\right)=\lfloor (n-1)/N_H\rfloor$. Let the BSs, IRSs, and UEs coordinates be denoted as $(x_s,y_s,z_s)$, $(x_l,y_l,z_l)$, and $(x_k,y_k,z_k)$. Accordingly, we set $sin\left({\psi }_{l,k}\right)=\left(z_k-z_l\right)/d_k^l$ and $sin\left({\vartheta }_{l,k}\right)cos\left({\psi }_{l,k}\right)=\left(y_k-y_l\right)/d_k^l$. Here, $d_k^l$ represents the distance between the k-th UE and the l-th IRS. Also, ${\mathit{a}}_{\mathrm{BS}}({\vartheta }_s,{\psi }_s)=[1,\dots ,e^{j\frac{2\pi (M-1)d^s}{{\lambda }_c}cos{(\vartheta }_1)cos\left({\psi }_1\right)}]$, where $d^s$ is the distance between adjacent antennas at the s-th BS. We set $cos\left(\vartheta s\right)cos\left({\psi }_s\right)=(x_l-x_s)/d_s^l$, where $d_s^l$ represents the separating distance between the the s-th BS and the l-th IRS. Likewise, the steering vector ${\mathit{a}}_{\mathrm{IRS}}({\vartheta }_{s,l},{\psi }_{s,l})$ can be computed, with $sin\left({\vartheta }_{s,l}\right)cos\left({\psi }_{s,l}\right)=\left(y_s-y_l\right)/d_s^l$, and $sin\left({\psi }_{s,l}\right)=\left(z_s-z_l\right)/d_s^l$. Note that reciprocity between downlink and uplink channels is assumed.

At the transmitting side, each BS transmits its signal with a beamforming vector to serve its associated UE. Hence, the transmitted signal by the s-th BS takes the form of:

$$x_s=\sqrt{p_s}{\mathit{w}}_s{s}_s,$$

(2)

where $p_s$ is the transmit power, ${{\mathit{w}}_s\in \mathbb{C}}^{M\times 1}$ denotes the beamforming vector, and the transmit signal, denoted as $s_s$, is characterized by random variables with a mean of zero and a variance of $\mathbb{E}\left[{\left|s_s\right|}^2\right]=1$, which holds for all UEs. We assume each BS applies the maximum ratio transmission (MRT) precoding based on both direct and indirect channels with its served UE for active beamforming design, i.e.,

$$\begin{array}{c}\hfill {\mathit{w}}_s=\frac{{{\mathit{H}}_k}^H}{\parallel {{\mathit{H}}_k}^H\parallel },\end{array}$$

(3)

where ${\mathit{H}}_k={\mathit{h}}_{s,dk}^H+{\sum }_{l\in \mathcal{L}}{\mathit{h}}_{rkl}^H{\Theta }_l{\mathit{G}}_{s,l}$ indicates the overall channel from the s-th BS to its served k-th UE; hence, $\mathbb{E}\left[{\left|x_s\right|}^2\right]=p_s$, i.e., $p_s\ge 0$.  

At the reflecting side, the l-th IRS adjusts the received signals from all BSs and subsequently reflects them. In particular, each n-th element of the l-th IRS reflects the incident signal, ${\widehat{x}}_{s,n}$, multiplied by a complex factor of reflection. Therefore, the mathematically defined expression for the adjusted reflected signal at the l-th IRS by the n-th element, resulting from the incident signal from the s-th BS, is as follows:

$$\begin{array}{c}\hfill {\widehat{y}}_{s,l,n}={\beta }_{l,n}{e}^{j{\theta }_{l,n}}{\widehat{x}}_{s,n},\forall s,l,n,\end{array}$$

(4)

where ${\beta }_{l,n}\in \left[0,1\right]$ and ${\theta }_{l,n}\in F_l=[0,2\pi )$ represent the amplitude and angle for the n-th element of the l-th IRS, respectively, and $F_l$ denotes the set of phase-shift values at each element. Note that in practical implementation, it is considered that each n-th element n-th of the l-th IRS is limited to a finite set of discrete values. Therefore, for discrete phase shifts in the l-th IRS, the angle of the n-th element is determined as ${\theta }_{l,n}\in F_l=\left\{0,\frac{2\pi }{2^b},\dots ,\frac{2\pi (2^b-1)}{2^b}\right\}$, where b denotes the number of bits used to determine the number of phase shift levels at each element. As such, the l-th IRS maps from the s-th BS incident signal vector to a corresponding reflected signal vector using the reflection beamforming matrix ${\Theta }_l$, which is the main characteristic for each IRS. Therefore, the reflected signal at the l-th IRS due to the s-th BS incident signal is:

$$\begin{array}{c}\hfill {\widehat{\mathit{y}}}_{s,l}={\Theta }_l\widehat{{\mathit{x}}_s},\forall s,l,\end{array}$$

(5)

where $\widehat{{\mathit{x}}_s}={[{\widehat{x}}_{s,1},\dots ,{\widehat{x}}_{s,N}]}^T$, and ${\Theta }_l=\mathrm{diag}({\beta }_{l,1}{e}^{j{\theta }_{l,1}},\dots ,{\beta }_{l,N}{e}^{j{\theta }_{l,N}})\in {\mathbb{C}}^{N\times N}$ and ${\widehat{\mathit{y}}}_{s,l}={[{\widehat{y}}_{s,l,1},\dots ,{\widehat{y}}_{s,l,N}]}^T$. We further set ${\beta }_{l,n}=1,\forall l,n$ in this system for maximizing the reflected signal by the IRSs [2,4].

At the receiving side, the k-th served UE $\in {\mathcal{K}}_s$ receives its desired signal both from its associated BS via the direct path as well as the indirect paths from the reflections and the scattering of the associated and non-associated IRSs, respectively. Considering the substantial path loss, only the first signal reflected by the IRSs is taken into account, while the others are ignored. Hence, the desired signal received at the k-th UE from the s-th BS is calculated as follows:   

$$\begin{array}{cc}\hfill y_{k,s}=& {\mathit{h}}_{s,dk}^H\sqrt{p_s}{\mathit{w}}_s{s}_s+\left(\sum _{l\in \mathcal{L}}{\mathit{h}}_{rkl}^H{\Theta }_l{\mathit{G}}_{s,l}\right)\sqrt{p_s}{\mathit{w}}_s{s}_s\hfill \\ \hfill =& {\mathit{h}}_{s,dk}^H\sqrt{p_s}{\mathit{w}}_s{s}_s+\underset{\mathrm{Associated}\mathrm{IRSs}}{\underbrace{\left(\sum _{l\in \mathcal{L}}{\lambda }_{l,s}{\mathit{h}}_{rkl}^H{\Theta }_l{\mathit{G}}_{s,l}\right)\sqrt{p_s}{\mathit{w}}_s{s}_s}}\hfill \\ \hfill & +\underset{\mathrm{Non}-\mathrm{associated}\mathrm{IRSs}}{\underbrace{\left(\sum _{l\in \mathcal{L}}(1-{\lambda }_{l,s}){\mathit{h}}_{rkl}^H{\Theta }_l{\mathit{G}}_{s,l}\right)\sqrt{p_s}{\mathit{w}}_s{s}_s}},\forall k\in {\mathcal{K}}_s.\hfill \end{array}$$

(6)

Meanwhile, the k-th UE is affected by co-channel interference from the $S-1$ non-associated BSs that serve the other $K-1$ UEs through direct and indirect links via all deployed IRSs. Thus, the total interference signal affecting the k-th UE is expressed as:

$${\ddot{I}}_k=\underset{\text{Inter-cell interference (co-channel interference)} I_k}{\underbrace{\sum _{i\in \mathcal{S},i\ne s}{\mathit{h}}_{i,dk}^H\sqrt{p_i}{\mathit{w}}_i{s}_i+\sum _{i\in \mathcal{S},i\ne s}\sum _{l\in \mathcal{L}}{\mathit{h}}_{rkl}^H{\Theta }_l{\mathit{G}}_{i,l}\sqrt{p_i}{\mathit{w}}_i{s}_i}}+\underset{\mathrm{Noise}}{\underbrace{n_k}},\forall k\in {\mathcal{K}}_s,$$

(7)

where $n_k$ is modeled as $\mathcal{CN}\left(0,{\sigma }_k^2\right),\forall k\in \mathcal{K}$. Therefore, the signal-to-interference-plus-noise ratio (SINR) observed at the k-th UE, associated with the s-th BS, is formulated as:

$${\mathrm{SIN}\mathrm{R}}_{k,s}=\frac{{\left|y_{k,s}\right|}^2}{{\left|I_k\right|}^2+{\sigma }_k^2},\forall k\in {\mathcal{K}}_s.$$

(8)

Then, the achievable rate of the k-th UE, expressed in bps/Hz, is determined as:

$$R_{k,s}=\sum _{s\in S}{\rho }_{k,s}{log}_2(1+{\mathrm{SINR}}_{k,s}),\forall k\in {\mathcal{K}}_s.$$

(9)

Accordingly, we define the system utility as the sum rate of all served UEs by:

$$R_{\mathrm{sum}}=\sum _{k\in {\mathcal{K}}_s}{R}_{k,s}=\sum _{k\in \mathcal{K}}\sum _{s\in \mathcal{S}}{\rho }_{k,s}{log}_2(1+{\mathrm{SIN}\mathrm{R}}_{k,s}).$$

(10)

3. Formulation of the Optimization Problem

The target of this work is the maximization of the system utility by maximizing the sum rate of the served UEs while still guaranteeing their rate constraints by jointly optimizing BS–UE associations $\left\{{\rho }_{k,s}\right\}$, BS–IRS associations $\left\{{\lambda }_{l,s}\right\}$, IRSs beamforming matrices ${\left\{{\Theta }_l\right\}}_{l=1}^L$, and BSs power control $\mathit{P}$. Therefore, we formulate the optimization problem as:

$$\begin{array}{cc}\hfill \left(\mathrm{P}1\right):\underset{\left\{{\rho }_{k,s}\right\},\left\{{\lambda }_{l,s}\right\},{\left\{{\Theta }_l\right\}}_{l=1}^L,\mathit{P}}{max}& R_{\mathrm{sum}}\hfill \end{array}$$

(11a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& R_{k,s}\ge R_{\min },\forall k\in {\mathcal{K}}_s,\forall s\in \mathcal{S},\hfill \end{array}$$

(11b)

$$\begin{array}{cc}\hfill & \sum _{s\in \mathcal{S}}{\rho }_{k,s}\le 1,\forall k\in \mathcal{K},\hfill \end{array}$$

(11c)

$$\begin{array}{cc}\hfill & \sum _{k\in \mathcal{K}}{\rho }_{k,s}\le 1,\forall s\in \mathcal{S},\hfill \end{array}$$

(11d)

$$\begin{array}{cc}\hfill & \sum _{s\in \mathcal{S}}{\lambda }_{l,s}=1,\forall l\in \mathcal{L},\hfill \end{array}$$

(11e)

$$\begin{array}{cc}\hfill & {\theta }_{l,n}\in F_l,\forall l,n,\hfill \end{array}$$

(11f)

$$\begin{array}{cc}\hfill & 0\le p_s\le p_{\max },\forall s\in \mathcal{S},\hfill \end{array}$$

(11g)

$$\begin{array}{cc}\hfill & {\rho }_{k,s}\in \{0,1\},{\lambda }_{l,s}\in \{0,1\},\forall l\in \mathcal{L},\forall s\in \mathcal{S},\forall k\in \mathcal{K}.\hfill \end{array}$$

(11h)

Note that (P1) is a challenging mixed-integer nonlinear programming (MINLP) problem due to its non-convex nature and the coupling between the objective function and constraints. The inclusion of binary variables ${\rho }_{k,s}$ and ${\lambda }_{l,s}$ introduces combinatorial aspects and NP-hardness. Generally, to optimally solve this problem, a relaxation conversion can be applied to transform the original non-convex problem into a convex formulation. To be specific, applying convex relaxation to the objective involves using the concave envelope, e.g., the exponential function, to approximate the logarithmic function. For the constraints, the problem has two types of non-convex constraints: the discrete constraints of passive beamforming and the binary constraints of association matrices. The discrete constraints can be relaxed by applying the semidefinite programming (SDP) technique by allowing continuous phase shifts. Meanwhile, the binary constraints can be replaced with continuous variables constrained between 0 and 1. Then, use a penalty function to enforce the binary nature. A convex optimization solver capable of handling mixed-integer and continuous variables, such as CVX [35], can solve the resulting convex-relaxed optimization problem. Based on our large-scale system, solving large-scale mixed-integer convex problems can be computationally demanding, especially with SDP relaxations, which can lead to computationally expensive solutions. However, with the discrete phase shift configuration and fixing BSs’ power, or when relaxed to discrete counterparts, the optimization can be solved optimally by enumerating all feasible possible combinations. Nevertheless, this approach incurs high complexity, which becomes prohibitive with a large number of BSs, UEs, and IRSs. Note that in case of the continuous configurations to the IRSs and considering constraint (11d), the phases can be determined by the instantaneous CSI, as the direct and indirect channels are aligned in phase to serve the associated UE [4]. Thus, the n-th phase shift of the l-th IRS is calculated as:

$${\theta }_{l,n}=-\angle h_{rkl,n}-\angle \left({\mathit{g}}_{s,l,n}^H{\mathit{h}}_{s,dk}\right),$$

(12)

where $h_{rkl,n}$ is the n-th element of ${\mathit{h}}_{rkl}^H$ and ${\mathit{g}}_{s,l,n}^H$ is the n-th row vector in ${\mathit{G}}_{s,l}$. Additionally, some strategies and algorithms have been applied to find sub-optimal solutions to association matrices with given other optimization variables. Specifically, the maximum reference signal received power (RSRP) association-based method is commonly used for designing BS–UE associations [36]. Let ${\mathit{\rho }}_s=[{\rho }_{1,s},{\rho }_{2,s},\dots ,{\rho }_{k,s}]$ represent the solution for BS–UE associations and the received powers from the s-th BS to UEs are denoted as ${\mathit{P}}_s^r=[P_{1,s}^r,P_{1,s}^r,\dots ,P_{k,s}^r]$, where $P_{k,s}^r={\left|y_{k,s}\right|}^2$. Accordingly, each BS $s^{\prime }$-th BS serves $k^{\prime }$-th UE, with $k^{\prime }=\underset{i}{argmin}{P}_{i,s^{\prime }}^r$. This rule needs a prior design for phase shifts of the IRS elements to determine the total received signals at the UEs from the indirect paths. Furthermore, sub-optimal solutions for BS–IRS associations can be obtained with convergent results using the successive refinement algorithm [14]. This iterative algorithm updates the associated BS for each deployed IRS successively, while keeping the associations of all other IRSs fixed until no further improvement in the defined performance can be achieved. Specifically, consider the BS-IRS association solution as ${\lambda }_l=[{\lambda }_{l,1},{\lambda }_{l,2},\dots ,{\lambda }_{l,S}]$, $l\in \mathcal{L}$. For a given design of BS-IRS association, represented by $\left\{{\lambda }_l\right\}$ for all L IRSs, let $R_{\mathrm{sum}}\left({\lambda }_l\right)$ denote the corresponding maximum sum rate. During algorithm iteration, each $l^{\prime }$-th IRS optimizes ${\lambda }_{l^{\prime }}$ successively, while the associated BSs for all other IRSs, denoted as ${\left\{{\lambda }_l\right\}}_{l\ne l^{\prime }}$, are fixed to maximize $R_{\mathrm{sum}}({\lambda }_{l^{\prime }};{\left\{{\lambda }_l\right\}}_{l\ne l^{\prime }})$. Enumerating the S BSs for the $l^{\prime }$-th IRS efficiently obtains the optimal solution to $l^{\prime }$. Subsequently, the associated BS for the $l^{\prime }$-th IRS is updated, and this process continues for the next IRS. Thus, this algorithm is a timely strategy since it needs to solve many iterations until there is no further improvement in its performance.

The challenges within our large-scale system stem from intricate interactions and dependencies among diverse entities, including BSs, UEs, and IRSs. These entities are intricately linked through associations, beamforming, and power control decisions, introducing complexity to the optimization of resources. Associations dictate which UEs are served by specific BSs and IRSs, while the beamforming matrices of IRSs impact signal propagation characteristics. The mobility of UEs and the non-convex nature of the optimization problem add to the complexity. Facilitating enhanced coordination and adaptability becomes crucial for efficient resource utilization and interference minimization amid dynamic network conditions. Enhanced coordination ensures harmony among network entities, while adaptability permits dynamic adjustments to configurations and strategies. To tackle these challenges, we propose the MADRL approach. In MADRL, each agent, representing a network entity, autonomously learns and refines its decision-making policy through collaborative learning. This approach not only ensures fair resource allocation, minimum rate requirements, and efficient power usage, but also enhances the overall adaptability and coordination of the network in dynamic conditions.

4. Proposed MADRL-Based Algorithm

In this section, we present the proposed MADRL-based algorithm. We begin by reformulating the optimization problem in the form of an MMDP. Subsequently, we detail the proposed algorithm, considering two configurations of the IRS. Finally, an analysis of the algorithm’s convergence and complexity is provided.

4.1. MAMDP Formulation of the Optimization Problem

To apply MADRL, the initial step involves transforming the optimization problem into an MAMDP format. The MAMDP typically comprises five essential elements for each agent: agent i, state $s_t^i$, action $a_t^i$, reward function $r_t^i$, and policy ${\pi }_t^i$ [37].

Given our studied system and formulated optimization problem, the behavior of all BSs and IRSs is modeled as a MADRL MMDP, which involves a set of agents (i.e., BSs and IRSs): $\mathcal{G}=\{\mathcal{S}\cup \mathcal{L}\}$. Thus, we have a total of states sets ${\left\{{\mathit{S}}^j\right\}}_{j=1}^{S+L}$, actions ${\left\{{\mathit{A}}^j\right\}}_{j=1}^{S+L}$, and rewards ${\{\mathit{R}}^j{\}}_{j=1}^{S+L}$. Under the MAMDRL, at each time step t, each agent receives an immediate observation or state $s_t^i\in {\mathit{S}}^i$ from the environment, where ${\mathit{S}}^i$ represents the potential set of states for agent i. In response to this state, each agent chooses an action $a_t^i\in {\mathit{A}}^i$, where ${\mathit{A}}^i$ denotes the set of possible actions for agent i. Subsequently, the environment updates the states to the next states $s_{t+1}^i$ and supplies a performance metric reward $r_t^i$ for that state–action pair $(s_t^i,a_t^i)$. The mapping from states to actions, governed by each agent’s local policy ${\pi }_t^i(s_t^i,a_t^i)$, signifies the probability of selecting action $a_t^i$ for the current state $s_t^i$. Over time, each agent strives to learn an optimal policy by maximizing cumulative rewards in the collaborative multi-agent environment. All the agents operate independently in a shared environment and then form joint action $a_t=\{a_t^1,a_t^2,\dots ,a_t^{S+L}\}\in \mathit{A}$, where $\mathit{A}=\{{\mathit{A}}^1,{\mathit{A}}^2,\dots ,{\mathit{A}}^{S+L}\}$. The subsequent state $s_{t+1}$ is updated from the current $s_t=\{s_t^1,s_t^2,\dots ,s_t^{S+L}\}$ providing the joint rewards $r_t=\{r_t^1,r_t^2,\dots ,r_t^{S+L}\}\in \mathit{R}$ to the joint state–action pairs. Accordingly, the joint optimal policy ${\pi }_t(s_t,a_t)=\{{\pi }_t^1(s_t^1,a_t^1),{\pi }_t^2(s_t^2,a_t^2),\dots ,{\pi }_t^{S+L}(s_t^{S+L},a_t^{S+L})\}$ is formed, which can maximize the long-term rewards received. The states, actions, rewards, and policies of our proposed MADRL-based approaches are described below.

• States: For each BS agent $j\in \mathcal{S}$, we define its state as the current active beamforming at time step t, ${\left({\mathit{w}}_j\right)}_t$, the previous transmit power at time step $t-1$, ${\left(p_j\right)}_{t-1}$, the previous BS–UE associations vector at time step $t-1$, $({\left\{{\rho }_{k,j}\right\}}_{k=1}^K{)}_{t-1}$, the previous BS-IRSs associations vector at time step $t-1$, ${\left({\left\{{\lambda }_{l,j}\right\}}_{l=1}^L\right)}_{t-1}$, and the current position to the served UE, ${\left(\left\{(x_k,y_k,z_k)\right\}\right)}_t,k\in {\mathcal{K}}_s$, which is mathematically given by:

  $$s_t^j=\left\{{\left({\mathit{w}}_j\right)}_t,{\left(p_j\right)}_{t-1},({\left\{{\rho }_{k,j}\right\}}_{k=1}^K{)}_{t-1},{\left({\left\{{\lambda }_{l,j}\right\}}_{l=1}^L\right)}_{t-1},{\left(\left\{(x_k,y_k,z_k)\right\}\right)}_t\right\}\in \mathit{S},$$

  (13)

  whose dimension is $2M+K+L+4$. The term ${\left({\mathit{w}}_j\right)}_t$, determined by the CSI, serves as a surrogate for the CSI in $s_t^j$ at time t to capture variations in wireless channel conditions. Note that directly integrating the channel estimates among all system nodes into the local observation may not be feasible due to the impracticality of handling high-dimensional feedback, while for each IRS agent $l\in \mathcal{L}$, we define its state as the previous passive beamforming vector at time step $t-1$, ${\left({\theta }_l\right)}_{t-1}$, which is defined as:

  $$s_t^l=\left\{{\left({\mathit{\theta }}_l\right)}_{t-1}\right\}\in \mathit{S},$$

  (14)

  whose dimension is $2N$.
• Actions: For each BS agent $j\in \mathcal{S}$, its action includes the transmit power, ${\left(p_j\right)}_t$, BS–UE association vector, $({\left\{{\rho }_{k,j}\right\}}_{k=1}^K{)}_t$, and BS-IRSs associations vector, ${({\left\{{\lambda }_{l,j}\right\}}_{l=1}^L)}_t$. However, in order to handle constraint (11e) and reduce the action space of the BS–IRS associations from $L·S$ to L, we shift the BS-IRSs associations as a subfunction to IRS agents, where each IRS has to choose one BS. Moreover, in order to reduce the infeasible BS–UE association actions and handle constraints (11c) and (11d) during training, a centralized DRL element for all BS–UE associations is trained at one BS, and then a copy is delivered to all BSs. Consequently, the action of each BS agent j is denoted by:

  $$a_t^j=\{{(p_j)}_t,{({\rho }_{k,j}^{\prime })}_t\}\in \mathit{A},$$

  (15)

  where ${({\rho }_{k,j}^{\prime })}_t$ is the index of the served k-th UE by the s-th BS. Correspondingly, the dimension is simply 2, and the action space of BS–UE associations equals K, while for each IRS agent $l\in \mathcal{L}$, we define its action as the current passive beamforming, ${\left({\mathit{\theta }}_l\right)}_t$, and BS-IRS association index element, ${({\lambda }_{l,j}^{\prime })}_t$, where ${\lambda }_{l,j}^{\prime }$ is the index of the chosen j-th BS for l-th IRS. For example, if at time step t, the agent IRS l chose to associate with the fourth BS, then ${\left({\lambda }_{l,4}\right)}_t=1$ in the BS–IRS associations matrix. The action of each IRS agent l is denoted by:

  $$a_t^l=\{{\left({\mathit{\theta }}_l\right)}_t,{\left({\lambda }_{l,j}^{\prime }\right)}_t\}\in \mathit{A}.$$

  (16)

  which outputs a dimension of $2N+1$ and an action space of BSs-IRS associations of S.
• Rewards: As the objective function of the optimization problem is to maximize the sum rate of served UEs, establishing a connection between the reward and the objective function drives the realization of the MMDP goal to augment long-term rewards. Noting that each agent with a separate reward may behave selfishly, therefore hindering the global optimal solution. Thus, we assume all agents have the same reward after the joint action is performed. The environment sets the instant reward for all cooperative agents in this framework as the total sum rate achieved. Note that all the constraints in (P1) are handled above, except the constraint (11b), which considers the minimum QoS requirements constraints. If the resulting joint action fails to satisfy the minimum data rates of the UEs, agents are penalized to encourage them to modify inappropriate actions. The reward function includes a penalty value, which is represented by the difference between the minimum achieved rate and the specified constraint rate. Thus, the agents will receive a negative penalty if the minimum QoS is not satisfied for the UEs. Otherwise, all agents will receive positive rewards represented by the total utility achieved. Here, we set such a penalty value instead of zero to limit the sparsity of the rewards and reflect how far the agents are from achieving the minimum QoS. Therefore, each agent $i\in \mathcal{G}$ can receive an instant reward, determined as follows:

  $$r_t^i=\left\{\begin{array}{cc}{\left(R_{\mathrm{sum}}\right)}_t,\hfill & \mathrm{if}{\left(R_{k,s}\right)}_t\ge R_{\min },\forall k\in {\mathcal{K}}_s,\forall s\in \mathcal{S},\hfill \\ -|{\left(R_k^{\min }\right)}_t-R_{\min }|,\hfill & \mathrm{if}{\left(R_{k,s}\right)}_t<R_{\min },\exists k\in {\mathcal{K}}_s,\forall s\in \mathcal{S},\hfill \end{array}\right.$$

  (17)

  where $R_k^{\min }$ represents the minimum achieved data rate of all the served UEs at the t-th time step.
• Policies: In the proposed MADRL framework, effective collaboration hinges on creating an optimal joint policy, denoted as ${\pi }_t(s_t,a_t)$. This joint policy plays a pivotal role in guiding collective decision-making among agents, directing them to choose actions that enhance cumulative rewards in the dynamic multi-agent environment. The process of learning and contributing to this joint policy is iterative and adaptive for each agent. In response to ever-changing environmental states, agents continually adjust their local policies by evaluating the outcomes of their actions based on observed states. Reinforcement signals aid this learning process, guiding agents toward actions that maximize system utility over the long term through reward-based feedback. Further sections will provide a closer look at how agents refine their local policies, contributing to the overall success of the collaborative MADRL framework.

4.2. The Implementation of the Proposed MADRL-Based Algorithm

In this subsection, the proposed MADRL-based algorithm frameworks are described in detail to solve the optimization problem under continuous and discrete phases of the IRSs. These proposed algorithms mainly consist of two phases, namely training and implementation. Each phase will be detailed below.

4.2.1. MADRL-Based Algorithm under Continuous Phases Case

Note that conventional one-agent-DRL-based approaches can be applied to solve the problem (P1) by exploring all possible actions. However, in this work, the problem has huge state and action spaces with both continuous and discrete optimization variables, posing challenges for applying a one-agent-based approach. Accordingly, we propose a cooperative MADRL-based framework to handle the challenges and solve the problem efficiently, where the multi-agent-based approach can work together to reduce the state and action spaces. Since the joint action includes both discrete and continuous actions, we apply different DRL algorithms to train the agents.

As shown in Figure 2, for each BS agent, we apply two DRL units: deep deterministic policy gradient (DDPG) to optimize transmit power allocation, ${\left(p_j\right)}_t,j\in \mathcal{S}$, and deep Q-network (DQN) to optimize BS–UE associations vector, $({\{{\rho }_{k,j}\}}_{k=1}^K{)}_t$. For each IRS agent, we apply two DRL units: DDPG for continuous phases, i.e., ${\left({\mathit{\theta }}_l\right)}_t$ and DQN for BSs-IRS associations vector, i.e., ${({\{{\lambda }_{l,j}\}}_{l=1}^L)}_t$. Note that, according to our system design, we have a closed form (12) to obtain continuous phases by knowing the associations’ designs; thus, we do not need DDPG units for the phases at the IRSs in the continuous case. In addition, we implement a centralized DDPG unit that copies the local action to each BS for controlling their transmit powers. Additionally, all the DRL algorithms are connected to replay memory to store all trained experiences, where a central control is employed to control the local and joint experiences.

The DQN algorithm integrates deep neural networks (DNNs) with Q-learning, where the DNNs are used to approximate the Q-function, replacing the tabular representation for the Q-value function. DQN consists of two networks, i.e., one is the main neural network called Q network with parameters ${\theta }_q$, which estimates the Q-value for the current state and action pair. The other is a target Q network with parameters ${\theta }_{q^{\prime }}$. Note that ${\theta }_q$ and ${\theta }_{q^{\prime }}$ have identical structures, but ${\theta }_{q^{\prime }}$ is used to estimate the Q-values of the next step state–action pair. Furthermore, the DQN algorithm stores all acquired experiences in the integrated replay memory. Figure 3a illustrates the architecture of the DQN algorithm. Meanwhile, the DDPG algorithm follows the DPG theorem, which learns a Q-function and a policy simultaneously. It can deal with continuous action spaces, addressing the restriction of the DQN network to discrete action spaces only. As illustrated in Figure 3b, DDPG consists of four networks, i.e., the actor-network with parameters ${\theta }_{\mu }$, the target actor-network with ${\theta }_{{\mu }^{\prime }}$, the critic-network with parameters ${\theta }_q$, and the target critic-network with ${\theta }_{q^{\prime }}$. Specifically, the DDPG algorithm trains the actor network to select the best actions given the current state, while the performance critic network estimates Q-value performance. Similar to DQN, the target networks mirror the structure of the training networks but have distinct parameters, and they utilize experience replay memory.

Given the structure of the DRL algorithms outlined above, we employ the proposed algorithm to solve (P1), as outlined in Algorithm 1. In the beginning, the integrated networks for all DDPG and DQN units are initialized, and all training hyper-parameters are set. Assuming the agents’ interactions within the environment are segmented into I episodes, each comprising a finite number of T steps. Then, at the start of every training episode, the algorithm resets the environment with different locations for the UEs and their corresponding channel gains. Subsequently, all channels linking network nodes are computed. We apply Additionally, random initialization is applied to the transmit powers, beamforming vectors, and association matrices. Accordingly, the state $s_t$ is defined and provided as input stimulation. The agents then provide feedback on the decisions of local actions related to the integrated DRL units at each agent to form the joint action. Then, the BSs’ transmit powers, BSs’ active beamforming, IRSs’ passive beamforming, and BS–IRS–UE association matrices are formulated. Thereby, the current joint reward is computed, and the subsequent state $s_{t+1}$ is defined. This information is stored as the joint experience $\{s_t,a_t,r_t,s_{t+1}\}$ to create the integrated buffer. This algorithm requires a minimum of $N_B$ experiences to initiate network updates and learning. Consequently, the preceding steps are reiterated $N_B$ times to establish the initial mini-batch, but this occurs exclusively in the first episode. Following the creation of $N_B$ experiences, the algorithm proceeds to the training phase. Note that the flow chart of Algorithm 1 is figured in Figure 4 to provide a visual representation of the algorithmic steps and facilitate better understanding.

At each time step t, the algorithm extracts local states, actions, and rewards from the chosen $N_B$ samples. Specifically, during the learning process, the action value function of each agent j is updated according to the Bellman equation as follows:   

$$\begin{array}{cc}\hfill Q^j(s_t^j,a_t^j)& \leftarrow Q^j(s_t^j,a_t^j)+\alpha \left(r_t^j+\gamma \underset{a^j}{max}{Q}^j(s_{t+1}^j,a^j)-Q^j(s_t^j,a_t^j)\right),\hfill \end{array}$$

(18)

where $\alpha$ denotes the learning rate. Therefore, each agent j obtains the optimal value function by solving:

$$\begin{array}{c}\hfill Q^{{*}^j}(s_t^j,a_t^j)={\mathbb{E}}_{s_{t+1}^j}\left[r_t^j+\gamma \underset{a^j}{max}{Q}^{{*}^j}(s_{t+1}^j,a_t^j)|s_t^j,a_t^j\right],\end{array}$$

(19)

where $(s_{t+1}^j,a_{t+1}^j)$ represents the next state–action pair for agent j.

With the DQN algorithm structure, each agent j with a current state $s_t^j$ selects its action $a_t^j$ using $ϵ$-greedy exploration. The agent either chooses a random action with a probability of $ϵ$ or selects an action for which the value function Q is greatest with a probability of $1-ϵ$.

$$a_t^j=\left\{\begin{array}{cc}\mathrm{random}\mathrm{action},\hfill & \mathrm{with}\mathrm{probability}ϵ,\hfill \\ \underset{a_t^j\in A_j}{\mathrm{argmax}}Q(s_t^j,a_t^j),\hfill & \mathrm{with}\mathrm{probability}1-ϵ.\hfill \end{array}\right.$$

(20)

Since the association designs and IRS reflection phases are shared by the system nodes, fully distributed operation for the agents without environment state sharing may be infeasible for a large-sized network. In our cooperative multi-agent framework, we adopt independent learning with state and reward sharing, where all agents form a joint action and calculate the reward in (17). With this joint response, a new state $s_{t+1}$ is formed, and a new experience {$s_t,a_t,r_t,s_{t+1}$} is added to the reply. A random sample of size $N_B$ is chosen from the replay to update the weights of the Q-network. Consequently, the target value function Q is given by:

$$y_t^j=\left\{\begin{array}{cc}{r}_t^j,\hfill & \mathrm{if}{s}_{t+1}^j\mathrm{is}\mathrm{terminal}\mathrm{state},\hfill \\ r_t^j+\gamma \underset{a_{t+1}^j\in A_j}{max}Q(s_{t+1}^j,a_{t+1}^j;{\theta }_{q^{\prime }}),\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(21)

By using gradient descent, the Q network of agent j can be trained by minimizing the following DQN loss function, expressed as:

$$L\left({\theta }_q\right)={(y_t^j-Q(s_t^j,a_t^j;{\theta }_q))}^2.$$

(22)

Next, a smoothing update is applied to the target critic parameters, gradually adjusting them to track the changes in the critic network. This process helps stabilize the learning and prevent rapid fluctuations in the target values, as follows:

$${\theta }_{q^{\prime }}\leftarrow {\tau }_{\mathrm{DQN}}{\theta }_q+(1-{\tau }_{\mathrm{DQN}}){\theta }_{q^{\prime }},$$

(23)

where ${\tau }_{\mathrm{DQN}}\ll 1$ represents the target smooth factor for the DQN algorithm. Subsequently, adjust the probability threshold $ϵ$ for selecting a random action according to the decay rate ${\alpha }_{\mathrm{decay}}$. At the end of each training step, if $ϵ$ exceeds the minimum value ${ϵ}_{\min }$, it is adjusted as follows:

$$ϵ=ϵ(1-{\alpha }_{\mathrm{decay}}).$$

(24)

Note that $ϵ$ is maintained from the end of one episode to the beginning of the next one, resulting in a gradual, uniform decrease over multiple episodes until it reaches ${ϵ}_{\min }$.

Algorithm 1 The proposed MADRL-based algorithm.

1: Initialize the power allocation DDPG units for all the BSs agents, including actor-network $\mu (s;{\theta }_{\mu })$, target actor network ${\mu }^{\prime }(s;{\theta }_{{\mu }^{\prime }})$, critic network $Q(s,a;{\theta }_{q_1})$, target critic network $Q^{\prime }(s,a;{\theta }_{{q_1}^{\prime }})$ with parameters ${\theta }_{{\mu }^{\prime }}={\theta }_{\mu }$ and ${\theta }_{{q_1}^{\prime }}={\theta }_{q_1}$. 2: Initialize the BS–UE associations DQN units, including Q-network $Q(s,a;{\theta }_{q_2})$, target Q-network $Q^{\prime }(s,a;{\theta }_{{q_2}^{\prime }})$ with parameters ${\theta }_{{q_2}^{\prime }}={\theta }_{q_2}$. 3: Initialize the passive beamforming DQN units for all the IRS agents, including Q-network $Q(s,a;{\theta }_{q_3})$, target Q-network $Q^{\prime }(s,a;{\theta }_{{q_3}^{\prime }})$ with parameters ${\theta }_{{q_3}^{\prime }}={\theta }_{q_3}$. 4: Initialize the BS–IRS associations DQN units, including Q-network $Q(s,a;{\theta }_{q_4})$, target Q-network $Q^{\prime }(s,a;{\theta }_{{q_4}^{\prime }})$ with parameters ${\theta }_{{q_4}^{\prime }}={\theta }_{q_4}$. 5: Initialize the learning rates ${\alpha }_{\mathrm{DQN}}$ and ${\alpha }_{\mathrm{DDPG}}$, discount factor $\gamma$, target smooth factors ${\tau }_{\mathrm{DQN}}$ and ${\tau }_{\mathrm{DDPG}}$ mini-batch size $N_B$ and an empty experience replay D. 6: for episode $i=1,2,\dots ,I$ do 7:      Reset the environment and acquire $(x_k,y_k,z_k),\left\{{\mathit{h}}_{s,dk}^H\right\}$, $\left\{{\mathit{G}}_{s,l}\right\}$, and $\left\{{\mathit{H}}_{rl}\right\},\forall s,l,k$. 8:      Initialize ${\left\{p_s\right\}}_{s=1}^S,{\left\{{\mathit{w}}_s\right\}}_{s=1}^S$, ${\left\{{\Theta }_l\right\}}_{l=1}^L$,$\left\{{\rho }_{k,s}\right\}$ and $\left\{{\lambda }_{l,s}\right\},\forall s,l,k$. 9:      Form initial state $s_1$ to all agents. 10:    for step $t=1,2,\dots T$ do 11:      for agent $j=1,2,\dots ,S+L$ do 12:         if $j\in \mathcal{S}$ then 13:           Power allocation DDPG unit selects transmit power, i.e., ${\left(p_j\right)}_t$. 14:           BS–UE association DQN unit selects ${\left({\rho }_{k,j}^{\prime }\right)}_t$. 15:           Form the action $a_t^j$ of the BS j as (15). 16:         end if 17:         if $j\in \mathcal{L}$ then 18:           Passive beamforming DQN unit selects ${\left({\mathit{\theta }}_l\right)}_t$. 19:           BSs-IRS association DQN units select ${\left({\lambda }_{l,j}^{\prime }\right)}_t$. 20:           Form the action $a_t^l$ of the IRS l as (16). 21:         end if 22:      end for 23:      Form the joint action $a_t$ by all agents. 24:      Form the BSs transmit powers ${\left({\left\{p_s\right\}}_{s=1}^S\right)}_t$, the BSs active beamforming ${\left({\left\{{\mathit{w}}_s\right\}}_{s=1}^S\right)}_t$, IRSs passive beamforming ${\left({\left\{{\Theta }_l\right\}}_{l=1}^L\right)}_t$ and BS–IRS–UE associated matrices, i.e., ${\mathit{\rho }}_t,{\mathit{\lambda }}_t$. 25:      Calculate the current joint reward $r_t$ using (17). 26:      Form the next joint state $s_{t+1}$ for all agents. 27:      Fill the buffer D by the experience $\{s_t,a_t,r_t,s_{t+1}\}$. 28:      for agent $j=1,2,\dots ,S+L$ do 29:         if update then 30:           Select from the buffer a $N_B$ experiences $\{s_i,a_i,r_i,s_{i+1}\}$. 31:           if $j\in \mathcal{S}$ then 32:              Calculate the target values of the DDPG units according to (26). 33:              Calculate the target values of the DQN units according to (21). 34:              Update every U steps: ${\theta }_{\mu }$ according to (28), ${\theta }_{q_1}$ according to (27), ${\theta }_{{\mu }^{\prime }}$ according to (29), and ${\theta }_{{q_1}^{\prime }}$ according to (30). 35:              Update ${\theta }_{q_2}$ according to (22), update ${\theta }_{{q_2}^{\prime }}$ according to (23). 36:           end if 37:           if $j\in \mathcal{L}$ then 38:              Set the value function target of DQN units as (21). 39:              Update ${\theta }_{q_3}$ according to (22), update ${\theta }_{{q_3}^{\prime }}$ according to (23). 40:              Update ${\theta }_{q_4}$ according to (22), update ${\theta }_{{q_4}^{\prime }}$ according to (23). 41:           end if 42:         end if 43:      end for 44:      Assign the updated state as $s_{t+1}$. 45:    end for 46: end for

The DDPG algorithm is employed for optimizing the transmit power of the BSs. In contrast to DQN, which employs an $ϵ$-greedy strategy, DDPG applies the DPG theorem to update the network parameters, estimating the current action from the current state. The action is evaluated using the deterministic policy of the actor network. To ensure effective exploration in the continuous action space during the DDPG training process, an exploration policy is formed by incorporating a defined noise into the DDPG action, as follows:

$$a_t^j=\mu (s_t^j;{\theta }_{\mu })+{\mathcal{N}}_t,$$

(25)

where ${\mathcal{N}}_t$ is the deployed noise model. Accordingly, the target value function for each agent j is calculated as follows:

$$y_t^j=\left\{\begin{array}{cc}{r}_t^j,\hfill & \mathrm{if}{s}_{t+1}^j\mathrm{is}\mathrm{a}\mathrm{terminal}\mathrm{state},\hfill \\ r_t^j+\gamma Q(s_{t+1}^j,\mu (s_{t+1}^j;{\theta }_{{\mu }^{\prime }});{\theta }_{q^{\prime }}),\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(26)

During training, the algorithm updates the critic network through the minimization of the critic loss function based on the selected mini-batch samples, as follows:

$$L\left({\theta }_q\right)=\frac{1}{N_B}\sum _{t=1}^{N_B}{(y_t^j-Q(s_t^j,a_t^j;{\theta }_q))}^2.$$

(27)

Following these, a policy gradient function is applied to update the actor network, involving the calculation of the derivative of the policy’s performance J with respect to ${\theta }_{\mu }$, as computed below:

$${\nabla }_{{\theta }_{\mu }}J=\frac{1}{N_B}\sum _{t=1}^{N_B}\underset{G_{a_t}}{\underbrace{{\nabla }_{a^j}Q(s_t^j,\mu (s_t^j;{\theta }_{\mu });{\theta }_q)}}|\underset{G_{{\mu }_t}}{\underbrace{{\nabla }_{{\theta }_{\mu }}\mu (s_t^j;{\theta }_{\mu })}},$$

(28)

where $G_{{\mu }_t}$ and $G_{a_t}$ denote the gradient of the actor output and the critic output concerning the actor parameters and the actor action, respectively. Next, a smoothing mechanism is adopted to update the target networks, expressed as follows:

$${\theta }_{{\mu }^{\prime }}\leftarrow {\tau }_{\mathrm{DDPG}}{\theta }_{\mu }+(1-{\tau }_{\mathrm{DDPG}}){\theta }_{{\mu }^{\prime }},$$

(29)

$${\theta }_{q^{\prime }}\leftarrow {\tau }_{\mathrm{DDPG}}{\theta }_q+(1-{\tau }_{\mathrm{DDPG}}){\theta }_{q^{\prime }},$$

(30)

where ${\tau }_{\mathrm{DDPG}}\ll 1$ is the smooth factor for the target networks in the DDPG algorithm.

4.2.2. MADRL-Based Algorithm under Discrete Phases Case

Due to the practical implementation and hardware constraints, the n-th element of the l-th IRS generally takes a finite set of discrete values. Considering the discrete case of IRS phase shifts, we propose a cooperative MADRL-based framework under the discrete case in this subsection. The BS agents are similar to the case with continuous cases, where each BS agent $j\in \mathcal{S}$, adopts a DDPG unit for power allocation, ${\left(p_j\right)}_t$, and a DQN unit for BS–UE associations, $({\{{\rho }_{k,j}\}}_{k=1}^K{)}_t$. However, for the IRS agents, each IRS agent $l\in \mathcal{L}$, adopts two DQN units, i.e., one to handle the discrete phase shift actions, i.e., ${\left({\mathit{\theta }}_l\right)}_t$, and the other for the BSs-IRS associations vector, i.e., ${({\{{\lambda }_{l,j}\}}_{l=1}^L)}_t$, as shown in Figure 2.

During the implementation stage, the cooperative agents observe the environmental state and make decisions regarding the powers of the BSs, phases of the IRSs, and the configuration of associations based on the trained models. After that, they observe the updated state and make another decision. In this setting, we assume that the locations of the UEs are accessible to the BSs agents and then shared with IRSs agents; for example, predicting the location information involves processing the signals and channels during the uplink communication, including path loss coefficients, the angle of arrival (AoA), the angle of departure (AoD), and the time of arrival (ToA) [38]. The proposed MADRL-based algorithm integrates the contents of Section 4.2.1 and Section 4.2.2, and is summarized in Algorithm 1.

4.3. Analysis of the Proposed MADRL-Based Algorithm

In this subsection, we analyze the convergence and computational complexity of the proposed cooperative MADRL-based algorithm under continuous and discrete phases. Then, the implementation complexity is compared to benchmark schemes.

4.3.1. Convergence Analysis

The proposed cooperative MADRL-based algorithm integrates DQN and DDPG algorithms, which learn policies independently. Therefore, the convergence of the proposed algorithm relies on the convergence of the DDPG and DQN algorithms. Initially, we analyze the convergence of traditional Q-learning-based algorithms, as the DQN and DDPG algorithms are considered evolutionary extensions of Q-learning algorithms. According to [39], the Q-learning-based algorithm converges to the optimal Q-function when the learning rate guarantees $0\le {\alpha }_t\le 1$, ${\sum }_t{\alpha }_t=\infty$, and ${\sum }_t{\alpha }_t^2<\infty$. Moreover, consider $I_m$ as the m-dimensional unit hyper-cube ${[0,1]}^m$, and $\phi \left(t\right)$ is a continuous function that is bounded, non-constant, and monotonically increasing. Additionally, $C\left(I_m\right)$ denotes the space of continuous functions on $I_m$. For any provided function $f\in C\left(I_m\right)$ and $ϵ>0$, there exists a $F\left(x\right)={\sum }_{i=1}^N{v}_i\phi ({\mathit{w}}_i^{T}x+b_i)$ with real vector ${\mathit{w}}_i\in R^m,(i=1,\dots ,N)$, real constants $v_i,b_i\in R$, and integer N to obtain an approximation of the function $f^{\prime }s$ realization. Note that f is independent of $\phi$ and guarantees $|F\left(x\right)-f\left(x\right)|<ϵ,\forall x\in I_m$. In essence, functions represented by the form $F\left(x\right)$ are densely distributed within $C\left(I_m\right)$. Following the Stone–Weierstrass theorem [40], a large neural network and the proper choice of initial conditions can be used to get close to any non-linear continuous function. Thus, it can be concluded that the DQN algorithms converge to optimal solutions by a large-size neural network and appropriately tuned free parameters and initial parameters. For the DDPG algorithm, as it applies a deterministic policy gradient theory, it shows convergence to sub-optimal solutions [41]. In summary, the proposed cooperative MADRL-based algorithm is guaranteed to converge under both continuous and discrete phases.

4.3.2. Computational Complexity Analysis

The computational complexity of the proposed algorithm can be split into two primary parts: forward propagation and backpropagation for action selection and training models [31], respectively.

• Action selection complexity: Let $|S_{\mathrm{in}}|$ denote the input dimension, n represent the number of nodes in the hidden layer, and $\left|A\right|$ indicate the output dimension. Specifically, $\left|A\right|$ denotes the size of the action space ${\mathit{A}}^j$ in the DQN algorithms and the dimension of the continuous action $a^j$ in the DDPG algorithms [28]. The complexity of a single decision for one three-layer DNN is approximately:

  $$\mathcal{O}\left(\right|S_{\mathrm{in}}|·n+|A|·n+n).$$

  (31)
  In our system, we have S BSs and L IRSs acting as agents, all sharing the same state. Thus, the states, actions, and related complexities under continuous and discrete cases can be defined as follows:

  $$|S_{\mathrm{in}}|=(S+L)·(2M+K+L+4+2N),$$

  (32)

  $$\left|A\right|=\left\{\begin{array}{cc}S·(1+K)+L·\left(S\right),\hfill & \mathrm{if}\mathrm{continuous}\mathrm{phases},\hfill \\ S·(1+K)+L·(B^{N}L+S),\hfill & \mathrm{if}\mathrm{discrete}\mathrm{phases},\hfill \end{array}\right.$$

  (33)

  $$\mathrm{Complexity}\approx \left\{\begin{array}{c}\begin{array}{c}\mathcal{O}\left(\right(S·(2M+2K+L+2N+5)\hfill \\ +L·(2M+K+L+2N+4+S)+1)·n),\hfill \end{array}\mathrm{if}\mathrm{continuous}\mathrm{phases},\hfill \\ \begin{array}{c}\mathcal{O}\left(\right(S·(2M+2K+L+2N+5)\hfill \\ +L·(2M+K+L+2N+B^{N}L+S+4)+1)·n),\hfill \end{array}\mathrm{if}\mathrm{discrete}\mathrm{phases},\hfill \end{array}\right.$$

  (34)

  where B represents the quantization level with $B=2^b$.
• Training process complexity: The computational complexity of backpropagation in the DQN and DDPG algorithms for fully connected DNN (FC-DNN) per training step is approximated as:

  $$\mathcal{O}\left(\right|S_{\mathrm{in}}|·|A|·n),$$

  (35)

  where $\left|A\right|$ is determined based on the types of actions. Thus, the training complexity of all DNNs is approximated as:

  $$\mathrm{Complexity}\approx \left\{\begin{array}{cc}\mathcal{O}(I·T·N_B·(S+L)·|S_{\mathrm{in}}|·|A|·n),\hfill & \mathrm{for}\mathrm{backpropagation},\hfill \\ \mathcal{O}(I·T·N_B·(S+L)·\left(\right|S_{\mathrm{in}}|+|A|+1)·n),\hfill & \mathrm{for}\mathrm{action}\mathrm{selection}.\hfill \end{array}\right.$$

  (36)
  Here, the total complexity is approximated as the backpropagation training complexity, which has a higher order than the action selection complexity.

4.3.3. Implementation Complexity Analysis

Although the training stage requires a significant amount of time for the agents, it is conducted offline using a high-performance computational server, thereby enhancing overall time efficiency. In contrast, the implementation stage is carried out online with parallel processing, emphasizing a time-saving approach during this phase of the process. Specifically, in the implementation stage, each agent operates independently in parallel, enabling the execution of individual actions without dependence on other agents. Additionally, the DRL elements within each agent function concurrently to contribute to the final joint action of that particular agent. Consequently, the overall implementation time of the entire MADRL algorithm is contingent upon the time taken by the DRL unit that consumes the maximum time.

Actually, the implementation run time of the DQN units in the proposed algorithm is more time-consuming than that of the DDPG units. The reason for this lies in the exploration strategy employed by DQN, where the epsilon-greedy strategy is used. During implementation, this exploration strategy may result in increased computation as the agent evaluates multiple Q-values for different actions, taking into account the exploration probability. In contrast, DDPG, being a deterministic policy gradient method, directly outputs a specific action without stochastic exploration during implementation. This deterministic nature simplifies the implementation process and can potentially lead to faster inference times. It is worth noting that we can reduce the complexity of the DQN units by dividing the function of each DQN into multiple parallel DQNs, for example, each DQN for each element or some elements. This approach to optimization could be explored in future work. Thus, we can compute the implementation complexity of the proposed algorithms under continuous and discrete phases as follows:

$$\mathrm{Complexity}\approx \left\{\begin{array}{cc}\mathcal{O}\left(\right(2M+2K+L+2N+5)·n),\hfill & \mathrm{if}\mathrm{continuous}\mathrm{phases},\hfill \\ \mathcal{O}((2M+K+L+2N+B^{N}L+S+5)·n),\hfill & \mathrm{if}\mathrm{discrete}\mathrm{phases}.\hfill \end{array}\right.$$

(37)

It seems that the complexity of the proposed algorithms is a polynomial function of $L,N,M,K,B,$ and n. Thus, we can represent the implementation complexity of the proposed algorithm as $\mathcal{O}\left(P\right(L,N,M,K,n\left)\right)$ and $\mathcal{O}\left(P\right(L,N,M,K,B,S,n\left)\right)$, under continuous and discrete cases, respectively. In

Table 1. Implementation complexity comparison.

Approach	Complexity
Exhaustive Approach	$\mathcal{O}\left(S^{K+L}\right)$
Graph-theoretic Algorithm	$\mathcal{O}(K·S^2)$ [42]
Successive Refinement Algorithm	$\mathcal{O}(I_t·K·S^3)$ [13,14]
Proposed MADRL Algorithm (continuous)	$\mathcal{O}\left(P\right(L,N,M,K,n\left)\right)$
Proposed MADRL Algorithm (discrete)	$\mathcal{O}\left(P\right(L,N,M,K,B,S,n\left)\right)$

, we compare the computational complexities of the prior arts with our proposed one. From the table, it shows that the prior approaches have an exponential complexity, which is higher than the polynomial complexity of the proposed approach. Here, $I_t$ denotes the number of reached iterations until convergence of successive algorithm is met.

In

Table 2. Average implementation run-time (seconds) under continuous phases case.

	N	5	10	15	20
Algorithm
Successive Refinement Algorithm		44.6637	78.5485	120.3662	153.0799
Proposed MADRL Algorithm		0.4372	0.4380	0.4424	0.4669

and

Table 3. Average implementation run-time (seconds) under discrete phases case.

	N	5	10	15	20
Algorithm
Successive Refinement Algorithm		80.8549	94.6200	101.2413	134.2253
Proposed MADRL Algorithm		0.4478	1.2505	38.9429	1050

, we assess the proposed average implementation run time MADRL-based algorithm under both continuous and discrete phases in comparison to the successive refinement algorithm for different numbers of reflecting elements under both continuous and discrete phase configurations, respectively. The average implementation run time is calculated from online predictions and averaged through 100 random initializations. These results are conducted on a server-grade machine featuring an Intel(R) Xeon(R) Silver 4110 CPU with a clock speed of 2.10 GHz and equipped with 16 GB of RAM. Specifically, under continuous phases, the overall implementation time of the entire MADRL algorithm is contingent upon the time taken by the DQN units for BS–UE associations at the BS agents, which consumes the maximum time. Meanwhile, under discrete phases, the overall implementation time of the entire MADRL algorithm is contingent upon the time taken by the DQN units to configure the discrete phases at the IRS agents. Additionally, it is observed that the complexity of the proposed algorithm under the discrete case is higher than in the continuous case, primarily due to the increasing number of elements, which amplifies the complexity of the DQN units. However, when compared to iterative approaches, the proposed algorithm demonstrates efficiency in run-time. As indicated in the tables, the proposed MADRL-based algorithm has a lower average prediction run time than the successive refinement algorithm, which necessitates solving many sub-iterations alternately until the solution converges.

5. Simulation Results

In this section, we present the numerical results for the proposed algorithm under the continuous phases scenario and the discrete phases scenario. For the system parameters regarding the network deployment, we follow the latest relevant works in [14]. Figure 5 depicts the 2D deployment of the studied system, where four BSs are positioned by coordinates $(x_s,y_s,z_s)$ at (−30,4,5) [m], (−30,−4,5) [m], (30,−4,5) [m], and (30,4,5) [m], respectively. Additionally, six IRSs are positioned by coordinates denoted by $(x_l,y_l,z_l)$ as follows: (−5,−10.5,5) [m], (−2.5,−12,5) [m], (2.5,−11.5,5) [m], (−5,−10.5,5) [m], (−2.5,−12,5) [m], and (2.5,11.5,5) [m], respectively. These IRSs share the same BS height. A total of six UEs are distributed randomly along the circumference of a circle. Specifically, the circle is centered at (0,0) with a diameter of $D=20$ [m] and positioned in front of the IRSs. We assume the UEs change their positions at the beginning of each episode. Following a random walk, each UE has a moving angle $g_k$ ($0\le g_k\le 2\pi$) and a moving speed $v_k$ ($0\le v_k\le 1)$ [m/s] within the confined region, as depicted in Figure 5. In the simulations, we set $2d^l=2d^s={\lambda }_c$, $N_H=1$ for odd N, $N_H=2$ for even N, and increase $N_V$ at the IRSs. The path losses in the channels are expressed by the models $32.6+36.7log10\left(d^{\mathrm{direct}}\right)$ [dB] [8] and $35.6+22log10\left(d^{\mathrm{indirect}}\right)$ [dB] [8], respectively. Here, $d^{\mathrm{direct}}$ represents the distance between the s-th BS and the k-th UE in the direct link, while $d^{\mathrm{indirect}}$ signifies the corresponding distance between nodes in indirect links, namely the s-th BS to the l-th IRS or the l-th IRS to the k-th UE. Other simulation parameters include $\epsilon$ = 10, noise power ${\sigma }_k^2=-80$ [dBm] and $R_{\min }$ = 1 [bps].

In the proposed cooperative MADRL framework, all DRL units, including DDPG and DQN units, are structured from FC-DNNs. Each network is composed of five layers, i.e., one input layer, one output layer, and two hidden layers, with a normalization layer in between to expedite convergence and reduce training time. The dimensions of the networks are contingent on the sizes of states and actions. We configure the hidden layer to be suitable for our studied system setup [21]. This configuration strikes a balance between model complexity and performance, as confirmed through empirical testing. In addition, Tanh and Relu models are applied as activation functions, chosen for their facilitation of gradient descent and backpropagation. To explore the action set in the DDPG algorithm, we introduce a complex Gaussian noise with zero mean and variance, i.e., ${\sigma }^2$. Denote $T_s$ as the sampling time; accordingly, ${\sigma }^2$$·\sqrt{T_s}$ typically satisfy $(1\%\mathrm{to}10\%)$ of the range of the DDPG action. The remaining hyper-parameters are listed in

Table 4. Proposed MADRL framework hyper-parameters settings.

Hyper-Parameter	Setting
Learning rates ${\alpha }_{\mathrm{DQN}}$, ${\alpha }_{\mathrm{DDPG}}$	0.001
Discount factor $\gamma$	0.95
Buffer size C	200,000
Mini-batch size $N_B$	16
Probability threshold $ϵ$	1
Min Probability threshold ${ϵ}_{\min }$	0.01
Decay rate ${\alpha }_{\mathrm{decay}}$	0.0050
Target smooth factors ${\tau }_{\mathrm{DQN}},{\tau }_{\mathrm{DDPG}}$	0.001
Target update frequency U	1
Sampling time $T_s$	1

. Part of these parameters is aligned with [21,31], while others are chosen through extensive trials.

To demonstrate the performance of the proposed algorithm, we compare it with benchmark schemes. The scenarios and benchmark schemes under consideration include:

• Without IRSs: conventional system without IRSs aid, i.e., switch off all the deployed IRSs and no related associations.
• Random setup: system with the BS–IRS–UE associations are set up randomly.
• Nearest setup: system with the BS–IRS–UE associations are set up based on the nearest association rule.
• Successive refinement-based algorithm: system with the BS–IRS associations are set up based on the iterative successive refinement optimization algorithm [13,14].

5.1. Performance Analysis of the Proposed MADRL-Based Algorithm

The proposed cooperative MADRL-based algorithm should be trained to yield high sum rates and output performance. Consequently, we monitor the network during episode steps’ training and investigate the effects of the learning rates and target smooth factors for both DQN and DDPG units. These parameters are considered critical hyper-parameters for the agents. It is important to note that we initially consider the continuous phases case to tune these parameters and then apply the tuned values to the discrete phases case.

Figure 6 plots the average reward versus the episode steps under various learning rates for the agents’ learning units. The average rewards are computed by accumulating instantaneous rewards over episode steps. It is observed that, for any given learning rate, agents initially receive low rewards. As the number of steps increases, these rewards grow until convergence to the episode occurs. This is primarily due to the fact that the relatively lower experience in the buffer during the early steps leads to a higher sample correlation and, consequently, poorer training performance for the agents. However, as experiences accumulate in the buffer, the agents collaborate to enhance the long-term rewards. Specifically, the figure illustrates that the curve attains the highest average rewards at a learning rate of 0.001. The improvement in performance resulting from the lower learning rate comes with the drawback of an extended convergence time. Additionally, it is worth noting that higher rates lead to increased oscillations, resulting in performance degradation. This implies that an optimal learning rate is necessary, as excessively low or high values may compromise both performance and convergence. Thus, we configure the learning rates for both DQN and DDPG units in all the agents to be 0.001, ensuring the optimization of long-term rewards and aligning with the main target of maximizing the system sum rate.

We also fine-tune the smoothing factor used in both DQN and DDPG units. Typically, it is necessary to set this value to less than 1 to ensure smoothness. In Figure 7, the average rewards are plotted against episode steps across various smooth factors. Observations reveal that utilizing a smooth value of 0.001 produces superior performance when compared to alternative values. Furthermore, it is observed that an increase in this value diminishes average rewards and degrades overall performance. This suggests that employing low smooth values for the agents is well-adapted for updating the target networks and converging towards optimal solutions for the studied system. Thus, we employ a smooth factor of 0.001 for both the DQN and DDPG units in all agents.

Figure 8 depicts the average total rewards achieved during total episodes with the adjusted parameters under both continuous and discrete cases. This average is determined by summing the cumulative values across all episodes. At the beginning of each episode, it is important to note that the environment undergoes a reset to assign new channel gains and UE positions. We assume a random walk for UEs, where each UE has a moving angle $g_k$ ($0\le g_k\le 2\pi$) and a moving speed $v_k$ ($0\le v_k\le 1)$ [m/s] within the confined region, as depicted in Figure 5. The plot demonstrates that the average total achieved sum rate increases with the number of training episodes until convergence is achieved. This indicates that the agents update their policy after each episode, progressively optimizing toward the objective of maximizing the system sum rate under varying positions for UEs with different channel gains.

5.2. Performance Comparison of the Proposed MADRL-Based Algorithm under the Continuous Phases Case

In this subsection, we compare our proposed cooperative MADRL-based algorithm under the case of the continuous phases with the presented benchmark schemes regarding the optimized total sum rate of the served UEs.

Figure 9 plots the optimized sum instantaneous sum rate versus the BS maximum transmit power under different schemes, where UEs’ locations are generated from a single random realization, as illustrated in Figure 5. Note that all the benchmarks are enumerated on five discrete power levels to form a power space in the range $P_d=\left\{0,\dots ,p_{\max }\right\}$.

From Figure 9, we note that our cooperative MADRL-based algorithm achieves the highest optimized sum rate compared to other benchmark schemes. This underscores the superiority of the proposed algorithm in reaching optimized values for the sum rate. The figure shows that the optimized sum rate is improved by increasing the BS maximum transmit power $p_{\max }$ until it reaches a limit, where the power range of BS is consistent with [4,8,23,31]. Similar to a traditional network without IRSs-aid, this behavior arises from the potential increase in transmit power from the BSs, boost impacts the strength of both the desired signal and co-channel interference at the served UEs in a comparable manner. Consequently, the SINR experiences enhancement until reaching a limit as $p_{\max }$ grows larger. In contrast, it is noted that with any given $p_{\max }$, well-optimizing the BS–IRS–UE associations can dramatically improve the optimized rate beyond the converged level. Contrasted with scenarios incorporating IRSs, the total achieved rate for the scenario without IRSs is the lowest because this scenario does not have the power gain due to the deployment of IRSs. In the conventional system without IRSs, the associations between BSs and UEs are set based on the nearest association rule. Then, the performance of the random setup scheme ranks as the second-worst. This indicates the importance of optimizing the BS–IRS–UE associations rather than randomly selecting them to maximize the benefits of IRS deployment. Applying the nearest association rule between system nodes enhances the sum rate compared to random associations. Moreover, when applying the successive refinement algorithm for optimizing the BS–IRS association, considering the BS–UE association based on nearest and RSRP setups, we can notice that the performance of the successive refinement algorithm-based RSRP is better than the successive refinement algorithm-based nearest. This is because, for any given configuration of IRS phases, the maximum RSRP association rule is a CSI-based association strategy that is determined by the strength of the UE received signal from both direct and indirect paths instead of distance-based association in the nearest setup. This is also well suited to the MRT active beamforming design in our system model. Concluding, we can find improved optimizations by applying the proposed algorithm, which jointly optimizes all optimization variables for maximizing the long-term sum rate.

Figure 10 plots the optimized BS–IRS–UE associations under different approaches for maximizing the sum rate at $p_{\max }$ = 4 dBm. At each time step, the served UEs are marked by the same color as their associated BSs and IRSs, while the waiting UEs or those not served at this time step are marked in “gray”. Figure 11 shows the corresponding achieved system utility by the serving BSs. In Figure 10a, without the aid of IRSs, each BS serves the nearest UE. This has the lowest network utility compared to approaches with the IRS aid, with UE 3 and 6 identified as bottleneck users. In Figure 10b, with IRSs activated and random associations between nodes, some gains are observed, reflecting the impact of setting up the BS–IRS–UE associations to maximize the total network utility. In Figure 10c, associating the nearest IRSs to aid the transmission between the served UEs and their serving BSs enhances the rates of the bottleneck UEs in Figure 10a and increases the total sum rate.

In Figure 10d, the successive refinement algorithm-based nearest algorithm optimizes the BS–IRS association to maximize the achieved utility. The BS1–IRS association is kept the same, as it already achieved a high rate by serving UE6. To maximize the sum rate for other BSs and increase the achieved rates at their served UEs, re-association is performed, as shown in Figure 10d. For example, BS4 is associated with two extra-aiding IRSs to aid the transmission of the UE5. In Figure 10e, by re-associating the BS–UE based on RSRP instead of the nearest approach, the successive refinement algorithm demonstrates that it can increase the BSs’ sum rates. This is because RSRP considers the direct and indirect channel conditions between BSs and UEs, instead of just the distance, as in the nearest approach case. For example, BS4 serves UE1 instead of UE5, maximizing the utility of BS4. Note that the BS–UE in the RSRP-based association is determined based on the random phases for the IRSs, which may affect the optimal BS–UE with the instant CSI. Figure 10f plots the optimized BS–IRS–UE associations by applying the proposed MADRL-based algorithm, which led to maximizing the total achieved rates, as shown in Figure 11. This reflects that the agents work together over time to improve the association matrices and, as a result, fine-tuning the phases and the BSs’ power controls to maximize the overall system rate.

Figure 12 plots the optimized total average sum rate versus the number of elements per IRS in the continuous case for various schemes. Here, we average each simulated result over 100 random UE position initializations following their mobility model as described above, with a transmit power constraint of 0 dBm for each BS. The plot shows that the average achieved rate for the proposed algorithm increases with the growing number of IRS reflecting elements, surpassing the iterative successive refinement algorithm. This indicates that increasing the number of reflecting elements, along with well-optimized association design and power control, enhances the gains brought by these IRSs, thereby improving the utility of the source BSs. Furthermore, the random and nearest setup algorithms achieve comparable results and, notably, their performance degrades as N increases. This degradation is due to their static association design, which relies on the fixed locations of UEs.

5.3. Performance Comparison of the Proposed MADRL-Based Algorithm under the Discrete Phases Case

In this subsection, we compare our proposed cooperative MADRL-based algorithm under the case of the discrete phases with the considered benchmark schemes in terms of the total sum rate of the served UEs.

Figure 13 plots the optimized sum instantaneous achieved rate for the served UEs versus the BS maximum transmit power under different schemes. As shown in Figure 13, the proposed cooperative MADRL-based algorithm allows the achieved sum rate to reach high values compared with other approaches, indicating the effectiveness of the proposed algorithm. Similar to Figure 9, under the case of the discrete phases, a system not relying on assistance from IRSs results in the lowest achieved rate. Moreover, the achieved rate of applying random associations between system nodes is still constrained by the association design, which is enhanced when applying the nearest association setup algorithm. Finally, the successive refinement algorithm shows sub-optimal solutions that depend on the iterative process. In the benchmark schemes, it is important to highlight that the discrete phase shifts are determined by quantizing the continuous phase shifts obtained to their nearest values in $F_l$. This quantization is expressed as ${\theta }_{l,n}^{*}=\underset{\theta \in F_l}{argmin}\left|{\theta -\theta }_{l,n}^{*}\right|$, where ${\theta }_{l,n}^{*}$ is the optimal discrete phase shift of the n-th element at the l-th IRS.

Under the case of the discrete phases, Figure 14 depicts the optimized BS–IRS–UE associations under both the successive refinement and the proposed cooperative MADRL algorithm designed to maximize the sum rate. Note that we omit association figures for other approaches, since they are the same as those in Figure 10. Figure 15 illustrates the corresponding achieved sum rate by the serving BSs. As depicted, applying the proposed cooperative MADRL algorithm results in a higher achieved sum rate at the BSs. This verifies the superiority of our proposed approach over other benchmarks.

Figure 16 plots the optimized total average sum rate versus the number of elements per IRS under the case of the discrete phase for the optimizing schemes. We average each simulated result over 100 random UE position initializations, with a transmit power constraint of 0 dBm for each BS. Similar to Figure 12, it is observed that the average achieved rate for the proposed algorithm increases with the growing number of the IRSs reflecting units and exhibits the highest achieved sum rate compared to other schemes. The successive refinement algorithm shows the second-best performance. For comparison, the random and nearest setup algorithms also achieve comparable results, with minimal enhancement in their performance as N increases. This indicates that increasing the number of reflecting elements with well-optimized association design and power control enhances the gains brought by these IRSs, thereby helping the BSs improve the total sum rate.

Figure 17 plots the optimized total average sum rate versus the number of elements per IRS under different resolutions for different schemes. In this context, we specifically consider both $b=1$ and $b=2$ for discrete phase shifts at each IRS. Similarly, we average each simulated result over 100 random UE position initializations, with a transmit power constraint of 0 dBm for each BS. From the figure, it is observed that as N increases, the performance gap between the considered schemes widens with increasing resolutions. This trend comes from the fact that higher resolutions provide more flexibility in designing the phases of the IRSs, leading to an enhancement in the total sum rate.

6. Conclusions

In this work, we have investigated a large-scale communication system with multi-BS assisted by multi-IRS to serve multi-UE, where the joint optimization of BS power control, IRS beamforming configuration, and associations design was studied to maximize the total sum rate. The cooperative MADRL-based algorithm was proposed for joint optimization under continuous and discrete phase cases. The demonstrated capability of these approaches includes real-time predictions in various channel conditions with different UE locations, while ensuring performance meets predefined criteria. The fine-tuning was applied to the proposed models, ensuring robust stability and convergence with a focus on maximizing long-term objectives. Our simulation results revealed that the proposed algorithm outperformed the presented iterative benchmarks with regard to the sum rate.