Flexible Resource Optimization for D2D XL-MIMO Communication via Adversarial Multi-Armed Bandit

Abstract

Extremely large-scale multi-input and multi-output (XL-MIMO) communication, compared to conventional massive multi-input multi-output communication, can support more users and higher data throughput, thereby significantly improving its spectral efficiency and spatial multiplexing capabilities. This paper investigates the optimization of resource allocation for device-to-device (D2D) multicast communication in XL-MIMO cellular networks. The “many-to-many” sharing model permits one subcarrier to be shared among multiple D2D groups (DGs) and each DG to reuse multiple subcarriers. The objective is to maximize the total multicast data rate of DGs while meeting the data rate requirements of cellular users. This optimization problem is formulated as a 0–1 mixed-integer nonlinear programming problem, with the challenge lying in the fact that adjusting the subcarriers and the power of the user equipment alters the network’s carrier occupation and interference relationships, thereby increasing computational complexity. To address this challenge, a phased strategy is proposed. Initially, subcarrier allocation and coarse power allocation are conducted for cellular users. Subsequently, an adversarial multi-player multi-armed bandit framework is employed, treating DGs as players and subcarrier and power combinations as arms, to maximize the total multicast data rate. An improved Exp3 algorithm is utilized for selecting the optimal combination of arms. Finally, precise power allocation for cellular users is conducted based on the allocation results of the DGs. A comparative analysis of various simulations confirms the superiority of our algorithm over the established heuristic subcarrier assignment and proposed power allocation (HSAPP) and the channel allocation scheme using full information of device locations (CAFIL) approaches.

1. Introduction

In the emerging era of sixth-generation (6G) mobile communication technology, the fast expansion of intelligent devices and the widespread adoption of mobile applications are giving rise to a vast amount of diverse content [1,2]. This requires wireless communication systems to provide higher data rates and connection densities [3]. In the exploration of 6G technology, the extremely large scale multi-input and multi-output (XL-MIMO) system has garnered considerable focus, primarily because of its capacity to significantly boost both spectral and energy efficiency [4]. This technology achieves precise signal control and interference management by deploying a high density of antennas at the base station (BS) and utilizing spatial multiplexing among users. This supports more user connections while improving the overall network throughput and maintaining signal quality [5,6]. Meanwhile, the demands of 6G are also driving the development of group-oriented services (namely, multicast and broadcast communication). Multicast communication takes advantage of the broadcast characteristics of wireless transmission, which can reduce data traffic pressure while simultaneously fulfilling the needs of multiple requesters [7].

When a group of users located in close proximity request the same content, conventional Device-to-Device (D2D) communication may result in redundant transmission by the BS, excessive signaling overhead, and bandwidth waste [8]. Therefore, multicast D2D (MD2D), which combines multicast technology and D2D communication, has been proposed [9]. It enables the extensive parallel processing of local content in high-density scenarios while facilitating a vast number of connections. In MD2D communication, resource management and scheduling strategies are primarily centered around two forms of multicast transmission: single-rate and multi-rate transmission [10]. The advantage of single-rate transmission lies in its simplification of resource management and scheduling strategies, ensuring that all users receive the same quality of service (QoS) and thereby improving the fairness and reliability of the system [11]. This paper takes single-rate MD2D XL-MIMO communication into account. In this case, the data rate achievable for each D2D group (DG) is restricted by the worst signal-to-interference-plus-noise ratio (SINR) condition of the receivers.

In MD2D communication systems, resource allocation models are categorized into three paradigms: one-to-one (exclusive subcarrier assignment with limited capacity [12,13,14]); one-to-many (single-subcarrier multi-DG sharing for improved spectral efficiency [15,16]); and many-to-many (M2M). The M2M model fundamentally outperforms its predecessors by enabling bidirectional subcarrier–DG multiplexing, where multiple subcarriers are dynamically shared among multiple DGs. This architecture achieves significant gains in spectral efficiency and resource flexibility while maintaining cellular user (CU) QoS. Advanced techniques such as a hybrid genetic algorithm for joint subcarrier power optimization [17] and a decentralized resource management framework [18] further enhance network throughput, guarantee minimum rate requirements, and ensure interference coordination, establishing M2M as a systematically optimized solution for high-density heterogeneous networks.

However, in refs. [17,18], the cellular BS only considers the use of MIMO or massive MIMO technologies. XL-MIMO expands the number of antennas and their deployment density comparted to that of massive MIMO to address future communication challenges. The near-field and far-field channel regimes exhibit fundamental distinctions in their wave propagation characteristics. In conventional massive MIMO systems operating under far-field assumptions, spatial stationarity and uniform plane wave propagation enable simplified channel modeling through angular domain approximations [19]. Conversely, XL-MIMO architectures with ultra-large apertures or high-frequency deployments enter the near-field regime, where spherical wavefronts introduce non-uniform phase variations across array elements and break spatial stationarity assumptions [20,21,22]. Currently, research on resource allocation for MD2D XL-MIMO communication is relatively scarce. The authors in [23] studied the joint subcarrier and power allocation problem in MD2D XL-MIMO communication, aiming to maximize the total multicast data rate of all DGs while ensuring QoS for the CUs, and proposed a heuristic subcarrier allocation algorithm and a power allocation algorithm based on differential concave functions, decomposing the original problem into two sub-problems and solving them separately. The aforementioned studies related to MD2D communication all assume the channel to be quasi-static. In practical applications, however, the dynamic changes in the position of user devices cause continuous variations in the channel information, which raise the requirements for the algorithm’s adaptability.

To accommodate dynamic variations in the environment, a feasible approach is to extract and exploit information from historical data for resource allocation. Fortunately, such an idea can be realized through reinforcement learning (RL). As a classic problem in RL, the multi-armed bandit (MAB) problem is characterized by its particular focus on balancing exploration and exploitation to maximize cumulative rewards over time. By employing classical strategies from the MAB framework, a communication network can adapt to changes in the environment effectively [24]. Recently, existing works have applied MAB strategies to 6G communication networks to efficiently utilize limited resources and provide a high quality for users [25]. Specifically, the authors in [26] proposed MAB-aided mode selection and addressed the challenge of adaptive relaying mode selection in 6G cooperative networks. To guarantee QoS for video streaming in highly dynamic mobile environments in heterogeneous 6G networks, the authors in [27] developed a QoS-driven contextual MAB framework integrated with multi-path quick user datagram protocol internet connection. On the other hand, the authors in [28] utilize adversarial MAB (AMAB) to cope with non-stationary environments during communication to improve network performance. Moreover, in [29], the authors introduced a “softwarization of intelligence” framework into 6G networks, which leverages MAB to dynamically select optimal policies tailored to real-time network states, and applied it to neighbor discovery and selection in D2D networks. In D2D network applications, classical MAB strategies have also been employed, with the aim of mitigating the impact of interference through efficient resource allocation and power control. Specifically, the authors in [30] utilize classical strategies, including the upper confidence bound and minimax optimal stochastic strategy, to design a neighbor discovery and selection algorithm in D2D networks, aiming to improve the average system throughput. In [31], the authors utilized a combinatorial MAB strategy to develop an integrated algorithm for mode selection and resource allocation in D2D networks. Compared to the aforementioned strategies, AMAB, as an unsupervised adaptive learning strategy, can employ dynamic adjustment methods to address changes and adversarial factors in the environment. It is more suitable for scenarios requiring real-time decision-making due to unpredictable environments. Consequently, AMAB can effectively handle channel variations and mobility among users and devices in D2D networks. The main works related to this strategy and their characteristics and configurations are listed and compared with those of the present paper in

Table 1. Summary of studies related to resource allocation in D2D networks.

Reference	Apply XL-MIMO	Support M2M	Dynamic Environment	Resource Allocation	Mode Selection	Neighbor Discovery	Apply RL
[12]				✓
[18]				✓
[23]	✓	✓		✓
[30]						✓	✓
[31]				✓	✓		✓
Present paper	✓	✓	✓	✓			✓

.

In light of the aforementioned issues, this paper considers optimizing the multicast data rate for all DGs, from the perspective of the entire cellular system, through subcarrier and power allocation, without affecting the QoS of CUs. However, adjusting the subcarriers and power for each user equipment (CU and DG) changes the carrier occupation relationships and interference relations within the network, making the direct solution to this optimization problem highly complex. To efficiently address this issue, this paper introduces an adversarial multi-player multi-armed bandit (MP-MAB) framework. The main contributions of this paper are as follows: (1) It establishes a resource allocation model for MD2D XL-MIMO communication. The goal is to optimize the aggregate multicast data rate across all DGs while adhering to the limitations on the minimum required data rate for CUs, the upper bound of transmission power, and the subcarrier occupation relationship between CUs and subcarriers. The formulated optimization problem is a 0–1 mixed-integer nonlinear programming (MINLP) problem, and it presents significant challenges for analytical solutions. (2) We model the original problem using the adversarial MP-MAB framework, where DGs are treated as players and the combinations of subcarriers and power for DGs and CUs are treated as arms. Since the arm space of this MAB problem is a four-dimensional (4D) space, with each arm having four variable parameters, exhaustively exploring all possible parameter combinations becomes impractical. Therefore, this study proposes a staged strategy to reduce the problem’s complexity. In the first stage, subcarrier allocation and rough power allocation for CUs are performed. Then, the adversarial MP-MAB framework is applied to solve the resource allocation problem for DGs, simplifying the arm space to two-dimensional (2D) space. In the second stage, based on the results of the D2D group allocation, precise power allocation for CUs is carried out. (3) Our simulation results demonstrate that the suggested algorithm performs effective subcarrier and power allocation. Comparisons with the HSAPP and CAFIL algorithms validate the effectiveness of the proposed approach.

2. System Model and Problem Formulation

Consider an MD2D communication system within a cellular network, consisting of a BS employing an extremely large-scale antenna array (with N elements), K single-antenna CUs, and D DGs, as illustrated in Figure 1. The commonly used boundary for distinguishing between the near-field and far-field regions is the Rayleigh distance, i.e., $Z=Q^2/\lambda$, where $\lambda$ is the carrier wavelength, $Q=N\varpi$ is the aperture of the antenna array, and $\varpi$ is the element spacing. For extremely large-scale antenna arrays, the Rayleigh distance may extend to several hundred meters. Therefore, this paper assumes that all user devices (CUs and DGs) are located in the near-field region. The system parameters are shown in

Table 2. System parameters.

Notation	Definition
N	Number of BS antenna
K	Number of CUs
D	Number of DGs
M	Number of receiving devices within DG
$\varpi$	Antenna element spacing of BS
Z	Rayleigh distance
${\mathbf{h}}_{k,0}$	Near-field channel from CU k to BS
$h_{d,m}$	Channel between transmitting device and receiving devices m within DG d
${\psi }_{k,0}^c$	Channel gain from CU k to BS on subcarrier c
${\psi }_{d,0}^c$	Channel gain from the transmitting device in DG d to BS on subcarrier c
${\psi }_{k,m}^c$	Channel gain from CU k to the receiving devices m in DG d on subcarrier c
${\psi }_{d,m}^c$	Channel gain from the transmitting device in DG d to receiving devices m on subcarrier c
$R_{\mathrm{req}}$	Required transmission rate of CUs
$p_k^{max}$	Maximum power threshold of CUs
$p_d^{max}$	Maximum power threshold of DGs

.

Assume the system has a total of C available orthogonal subcarriers, denoted by the index set $\mathcal{C}=\left\{1,2,\cdots ,c,\cdots ,C\right\}$. Similarly, let $\mathcal{K}=\left\{1,2,\cdots ,k,\cdots ,K\right\}$ and $\mathcal{D}=\left\{1,2,\cdots ,d,\cdots ,D\right\}$ represent the index sets for the CUs and DGs, respectively. For each DG d, its associated D2D receivers (DRs) are indexed by ${\mathcal{D}}_d=\left\{1,2,\cdots ,m,\cdots M\right\}$, where $\left|{\mathcal{D}}_d\right|$ denotes the number of DRs in the group. In the proposed MD2D framework, a D2D transmitter (DT) serves multiple DRs within the same group, while each DR belongs exclusively to one DG. This implies mutual exclusivity between groups: ${\mathcal{D}}_d\bigcap {\mathcal{D}}_{d^{\prime }}=⌀$. Given the spatial separation between DGs, each DG is uniquely served by a single DT. Furthermore, the spatial dynamics of the system ensure that DRs remain within the coverage of their assigned DT over time. Consequently, each DT and its associated DRs form a circular cluster.

The near-field channel between CU k and BS ${\mathbf{h}}_{k,0} {\in }^{N\times 1}$ can be expressed as follows [32]:

$${\mathbf{h}}_{k,0}=\sum _{\mathrm{s}=1}^S{\beta }_{k,0,s}\mathbf{a}\left({\mathcal{L}}_{k,0,s},{\theta }_{k,0,s}\right),$$

(1)

where S represents the total number of paths from CU k to the BS. ${\beta }_{k,0,s}$, ${\mathcal{L}}_{k,0,s}$, and ${\theta }_{k,0,s}$ denote the path gain, distance, and angle associated with path s of CU k, respectively. The steering vector $\mathbf{a}\left({\mathcal{L}}_{k,0,s},{\theta }_{k,0,s}\right)$ is expressed as follows:

$$\begin{array}{cc}\hfill & \mathbf{a}\left({\mathcal{L}}_{k,0,s},{\theta }_{k,0,s}\right)\hfill \\ \hfill & =\sqrt{{\displaystyle \frac{1}{N}}}{\left[e^{-j{\displaystyle \frac{2\pi }{\lambda }}\left({\mathcal{L}}_{k,0,s}^{\left(0\right)}-{\mathcal{L}}_{k,0,s}\right)},\cdots ,e^{-j{\displaystyle \frac{2\pi }{\lambda }}\left({\mathcal{L}}_{k,0,s}^{\left(N-1\right)}-{\mathcal{L}}_{k,0,s}\right)}\right]}^{\mathrm{T}}\hfill \end{array}$$

(2)

where ${\mathcal{L}}_{k,0,s}^{\left(n\right)}=\sqrt{{\mathcal{L}}_{k,0,s}^2+{\left(n\varpi \right)}^2-2nd{\mathcal{L}}_{k,0,s}sin{\theta }_{k,0,s}}$ represents the distance between the n-th BS antenna and CU k.

In DG d, the channel between DT and DR m can be represented as follows [12]:

$$h_{d,m}={\beta }_{d,m}\sqrt{{\mathcal{L}}_{d,m}^{-\alpha }},$$

(3)

where ${\beta }_{d,m}\sim \mathcal{C}\mathcal{N}\left(0,1\right)$ represents the path gain associated with small-scale fading, ${\mathcal{L}}_{d,m}$ represents the distance from DT to DR m, and $\alpha$ is the path loss exponent.

In an XL-MIMO system, the optimal precoding scheme can be designed via maximum ratio combining, which exploits the asymptotic orthogonality of channel vectors under the law of large numbers [33]. Specifically, the precoding vector is given by $\mathbf{f}={\mathbf{h}}^{\mathrm{H}}$. Consequently, the equivalent channel gain from CU k to the BS on subcarrier c can be represented as follows:

$${\psi }_{k,0}^c={\left|{\mathbf{h}}_{k,0}^c{\mathbf{f}}_{k,0}^c\right|}^2={∥{\mathbf{h}}_{k,0}^c∥}_2^2.$$

(4)

Similarly, the channel gain from the DT of DG d to the BS on subcarrier c is represented as ${\psi }_{d,0}^c$, the channel gain from CU k to the DR m of DG d on subcarrier c is represented as ${\psi }_{k,m}^c$, and the intra-group channel gain from the DT to the DR m within DG d on subcarrier c is represented as ${\psi }_{d,m}^c$.

Therefore, the received SINR on subcarrier c for CU k and DR m of DG d are, respectively, expressed as follows:

$${\gamma }_k^c={\displaystyle \frac{p_k^c{\psi }_{k,0}^c}{{\Gamma }_k^c+{\sigma }_0^2}},$$

(5)

$${\gamma }_{d,m}^c={\displaystyle \frac{p_d^c{\psi }_{d,m}^c}{{\Gamma }_{d,m}^c+{\sigma }_0^2}}.$$

(6)

The aggregate interference components are defined as ${\Gamma }_k^c={\displaystyle \sum _{d\in \mathcal{D}}}{\rho }_d^c{p}_d^c{\psi }_{d,0}^c$ (the interference from DGs) and ${\Gamma }_{d,m}^c={\rho }_k^c{p}_k^c{\psi }_{k,m}^c+{\displaystyle \sum _{d^{\prime }\in \mathcal{D},d^{\prime }\ne d}}{\rho }_{d^{\prime }}^c{p}_{d^{\prime }}^c{\psi }_{d^{\prime },m}^c$ (the interference from the CU and other DGs), where ${\rho }_k^c,{\rho }_d^c\in \{0,1\}$ are binary subcarrier allocation indicators for CU k and DG d, respectively, with $p_k^c$ and $p_d^c$ denoting their allocated transmit power on subcarrier c. The term ${\sigma }_0^2$ represents the additive white Gaussian noise variance.

To ensure fairness in resource allocation, the achievable data rate for DG d is computed based on the worst SINR among its receivers [13]. The worst received SINR for DG d on subcarrier c is given by

$${\gamma }_d^c=\underset{m\in {\mathcal{D}}_d}{min}{\gamma }_{d,m}^c.$$

(7)

Then, the achievable data rate for CU k and DG d on subcarrier c are, respectively, formulated as

$$R_k^c={log}_2\left(1+{\gamma }_k^c\right),$$

(8)

$$R_d^c=\left|{\mathcal{D}}_d\right|{log}_2\left(1+{\gamma }_d^c\right).$$

(9)

Hence, the total achievable rates for CU k and DG d are obtained by summing over all allocated subcarriers, i.e., $R_k={\sum }_{c\in \mathcal{C}}{\rho }_k^c{R}_k^c$ and $R_d={\sum }_{c\in \mathcal{C}}{\rho }_d^c{R}_d^c$.

The objective of this paper is to maximize the total multicast data rate of all DGs by jointly optimizing the subcarrier and power allocation while ensuring a minimum QoS of the CUs. Specifically, the optimization problem is formulated as

$$\begin{array}{cc}\hfill & \underset{\rho ,\mathbf{p}}{max}\sum _{d\in \mathcal{D}}{R}_d\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.{\mathrm{C}}_1:R_k\ge R_{\mathrm{req}},\forall k\in \mathcal{K}\hfill \\ \hfill & {\mathrm{C}}_2:\sum _{c\in \mathcal{C}}{\rho }_k^c{p}_k^c\le p_k^{max},\forall k\in \mathcal{K}\hfill \\ \hfill & {\mathrm{C}}_3:\sum _{c\in \mathcal{C}}{\rho }_d^c{p}_d^c\le p_d^{max},\forall d\in \mathcal{D}\hfill \\ \hfill & {\mathrm{C}}_4:\sum _{k\in \mathcal{K}}{\rho }_k^c\le 1,\forall c\in \mathcal{C}\hfill \\ \hfill & {\mathrm{C}}_5:p_k^c,p_d^c\ge 0,\forall k\in \mathcal{K},\forall d\in \mathcal{D},\forall c\in \mathcal{C}\hfill \\ \hfill & {\mathrm{C}}_6:{\rho }_k^c,{\rho }_d^c\in \left\{0,1\right\},\forall k\in \mathcal{K},\forall d\in \mathcal{D},\forall c\in \mathcal{C}\hfill \end{array}$$

(10)

where $R_{\mathrm{req}}$ is the required transmission data rate for the CUs. $p_k^{max}$ and $p_d^{max}$ are the maximum transmission powers for the CUs and DGs, respectively. The constraint ${\mathrm{C}}_1$ ensures the QoS of the CU, while the constraints ${\mathrm{C}}_2$, ${\mathrm{C}}_3$, and ${\mathrm{C}}_5$ are power limitations for the CUs and DGs. The constraint ${\mathrm{C}}_4$ stipulates that each subcarrier can be assigned to at most one CU, which corresponds to the definition in Equation (5). This optimization problem contains both 0–1 binary variables and continuous variables, making it difficult to solve directly.

3. Joint Allocation Algorithm Based on Improved EXP3

The optimization problem in Equation (10) requires joint power and subcarrier allocation for all DGs and CUs, which makes it a 0–1 MINLP problem [34]. To address this combinatorial complexity, we propose an adversarial MP-MAB framework with dimensionality reduction. Given the prohibitive exploration cost in the original 4D arm space, we implement a two-phase decomposition: In the first phase, subcarrier allocation and coarse power allocation are performed for the CUs, and then a joint subcarrier–power allocation for DGs is executed via an improved Exp3 algorithm that reduces the arm space to 2D. In the second phase, precise power allocation is carried out for the CUs. The flowchart of the proposed scheme is shown in Figure 2.

3.1. Problem Reformulation

The optimization problem in Equation (10) is viewed as an MP-MAB problem with the variable sets ${\mathcal{C}}_d$, ${\mathcal{P}}_d$, ${\mathcal{C}}_k$, and ${\mathcal{P}}_k$, where ${\mathcal{C}}_d$ and ${\mathcal{P}}_d$ are the candidate subcarrier set and power set for the DGs, while ${\mathcal{C}}_k$ and ${\mathcal{P}}_k$ are the candidate subcarrier set and power set for the CUs. In this MP-MAB problem, each DG d acts as an independent player whose action space ${\mathcal{A}}_d$ is defined as the Cartesian product ${\mathcal{C}}_d\otimes {\mathcal{P}}_d\otimes {\mathcal{C}}_k\otimes {\mathcal{P}}_k$, i.e., ${\mathcal{A}}_d={\left\{a_{d,i}\right\}}_{i=1}^{\left|{\mathcal{A}}_d\right|}$, where each arm $a_{d,i}$ represents a unique combination of subcarrier–power allocations. We discretize the time axis into slots with a time buffer T, during which user devices are assumed to be quasi-static (low-mobility). In each slot, players simultaneously select arms and observe the stochastic rewards determined by their choices.

The reward of DG d is defined as its normalized data rate, which can be expressed as

$$r_d\triangleq {\displaystyle \frac{R_d}{R_d^{*}}},$$

(11)

where $R_d^{*}={max}_{i\in \{1,2,\cdots ,\left|{\mathcal{A}}_d\right|\}}{R}_d\left(a_{d,i}\right)$ denotes the maximum observed rate across all possible arm selections. The cumulative total reward of DG d over T slots is defined as

$$r_d^{\mathrm{sum}}\triangleq \sum _{t=1}^T\sum _{i\in {\mathcal{A}}_{{\mathbf{a}}_d\left(t\right)}}{r}_{d,i}\left(t\right),$$

(12)

with ${\mathcal{A}}_{{\mathbf{a}}_d\left(t\right)}\triangleq \left\{i:A_{d,i}\left(t\right)\ne 0,1\le i\le \left|{\mathcal{A}}_d\right|\right\}$ representing the index set of the arms selected by DG d at slot t, where $A_{d,i}\left(t\right)\in \{0,1\}$ indicates the arm’s selection status. Each self-interested player (DG) aims to maximize $r_d^{\mathrm{sum}}$ through individual arm selection. However, reviewing the form of the data rate $R_d$ of DG d, a player’s choice of the subcarrier–power combination that maximizes their own $R_d$ may lead to a reduction in the $R_d$ of other players. This competitive interaction establishes an adversarial MP-MAB formulation. Given that the joint arm space ${\mathcal{A}}_d={\mathcal{C}}_d\otimes {\mathcal{P}}_d\otimes {\mathcal{C}}_k\otimes {\mathcal{P}}_k$ grows exponentially with the number of subcarriers and the power discretization’s granularity, full exploration becomes computationally prohibitive within a finite time buffer. To efficiently solve this MAB problem and reduce its complexity, a phased strategy with a dimension reduction mechanism must be developed.

3.2. Phase One: Resource Allocation for DGs

For the purpose of allocating subcarriers to CUs and DGs, an equal power allocation scheme is first applied to the CUs; that is, for $\forall k\in \mathcal{K}$, $p_k^c=p_k^{max}/C$. Following power initialization, subcarriers are assigned to the CUs through an iterative gap-reduction procedure. The rate deficiency metric for CU k is defined as

$$\Delta R_k=R_{\mathrm{req}}-R_k.$$

(13)

During each iteration, the CU with the maximal $\Delta R_k$ prioritizes subcarrier selection to minimize its rate deficit. This mechanism ensures the progressive satisfaction of all CUs’ QoS constraints (${\mathrm{C}}_1$). The complete procedure is formalized in Algorithm 1.   

Algorithm 1: Subcarrier allocation algorithm for CUs.

Therefore, after the iteration process is completed, the QoS of all CUs is guaranteed and the constraints ${\mathrm{C}}_1$, ${\mathrm{C}}_2$, and ${\mathrm{C}}_4$ are also adhered to. Thus, the original problem is transformed into

$$\begin{array}{cc}\hfill & \underset{\rho ,\mathbf{p}}{max}\sum _{d\in \mathcal{D}}{R}_d\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.{\mathrm{C}}_3:\sum _{c\in \mathcal{C}}{\rho }_d^c{p}_d^c\le p_d^{max},\forall d\in \mathcal{D}\hfill \\ \hfill & {\mathrm{C}}_5:p_d^c\ge 0,\forall d\in \mathcal{D},\forall c\in \mathcal{C}\hfill \\ \hfill & {\mathrm{C}}_6:{\rho }_d^c\in \left\{0,1\right\},\forall d\in \mathcal{D},\forall c\in \mathcal{C}.\hfill \end{array}$$

(14)

Next, we introduce the MAB framework to solve the optimization problem in Equation (14). The difference between this and Section 3.1 is that since the subcarriers and power of the CUs have already been allocated, the set of available arms ${\mathcal{A}}_d^{\mathrm{re}}$ for each player is simply the Cartesian product of the sets ${\mathcal{C}}_d$ and ${\mathcal{P}}_d$, denoted as ${\mathcal{A}}_d^{\mathrm{re}}={\mathcal{C}}_d\otimes {\mathcal{P}}_d$. The arm selection strategy employs an improved Exp3 algorithm to solve problem Equation (14) in a distributed manner [35].

First, the proposed algorithm accelerates convergence through prior knowledge integration. As indicated by Equation (18), the updated weight $w_{d,i}\left(t+1\right)$ depends on both the estimated reward ${\widehat{r}}_{d,i}\left(t\right)$ and the current weight $w_{d,i}\left(t\right)$. Additionally, the empirical probability mass function (PMF) of DG d, namely

$$u_{d,i}\left(t\right)=\left(1-{\alpha }_z\right){\displaystyle \frac{w_{d,i}\left(t\right)}{W_d\left(t\right)}}+{\displaystyle \frac{{\alpha }_z}{\left|{\mathcal{A}}_d^{\mathrm{re}}\right|}},$$

(15)

is a function of the current weights $w_{d,i}\left(t\right)$ and the exploration–exploitation factor ${\alpha }_z$, where $W_d\left(t\right)={\displaystyle \sum _{j=1}^{|{\mathcal{A}}_d^{\mathrm{re}}|}}{w}_{d,j}\left(t\right)$ represents the total arm weights. An adaptive ${\alpha }_z$ schedule is designed to balance exploration breadth and historical knowledge utilization: the initial stages employ larger ${\alpha }_z$ values to ensure sufficient arm exploration, followed by gradual decay to prioritize high-reward arm exploitation. The parameter is configured as follows:

$${\alpha }_z=min\left\{1,\sqrt{{\displaystyle \frac{\left|\left|{\mathcal{A}}_d^{\mathrm{re}}\right|\right|ln\left(\left|{\mathcal{A}}_d^{\mathrm{re}}\right|\right)}{\left(e-1\right){\vartheta }_z}}}\right\}.$$

(16)

By rationally setting the value of ${\vartheta }_z$, the empirical PMF can converge to the actual PMF. This adaptive mechanism enables rapid convergence to optimal subcarrier–power combinations. The complete distributed allocation procedure is formalized in Algorithm 2.

Algorithm 2 presents a method for finding a near-optimal solution to the above optimization problem. The algorithm starts by obtaining the initial weight of each arm in ${\mathcal{A}}_d^{\mathrm{re}}$. For this algorithm, we have no access to prior knowledge, so the initial weight of each arm is set to 1, at which point each arm has the same selection probability. The exact value of ${\vartheta }_z$ is given in line 3. Substituting ${\vartheta }_z$ into Equation (16) gives ${\alpha }_z=2^{-z}$, from which it can be observed that ${\alpha }_z$ decays exponentially. When z reaches a sufficiently high value, the latter component in Equation (15) approaches 0.

Starting from the internal loop in line 5, the selection probability of each arm is first calculated according to Equation (15), and then ${\varsigma }_t^d$ arms are selected from ${\mathcal{A}}_d^{\mathrm{re}}$ based on their probability. Since the algorithm can only observe the rewards for the arms selected in each round, it cannot directly observe the rewards of other arms. To better evaluate the potential value of each arm, the estimated reward is computed in line 9, which helps the algorithm take the unobserved rewards into account, thus avoiding premature convergence to suboptimal arms. In addition, updating the weights directly using the observed rewards leads to bias because the reward information for unselected actions is missing. Therefore, weights are updated using estimated rewards, as shown in line 11. Finally, the cumulative estimated rewards of the selected action are updated. The internal loop is repeated until the maximum cumulative reward is greater than the set threshold, and then the algorithm proceeds to the next external loop.

Finally, we briefly analyze the computational complexity of the proposed algorithm (Algorithm 2). The computational complexity of the arm selection strategy for the adversarial MP-MAB formulation mainly depends on the implementation of the arm selection strategy. In its basic implementation, multiple trials are required to estimate the value of each arm. Given that the computational complexity of calculating the selection probability for each arm and the complexity of selecting an arm based on the probability are both $O\left(\left|{\mathcal{A}}_d^{\mathrm{re}}\right|\right)$, the complexity of Algorithm 2 is of the first order, i.e., $O\left(\left|\mathcal{D}\right|\left|{\mathcal{A}}_d^{\mathrm{re}}\right|T\right)$.

Algorithm 2: Joint subcarrier and power allocation algorithm based on improved Exp3 and DG d.

3.3. Phase Two: Greedy Power Reallocation for CUs

After the DGs complete their subcarrier and power allocation, power is reassigned to the CUs. According to Equation (5) and the constraint ${\mathrm{C}}_4$, there is no interference between the CUs, and interference only originates from the DGs.

Based on the power of DG d, which shares subcarriers c with CU k, the lower bound of the power ${\stackrel{⌣}{p}}_k^c$ for CU k that satisfies the data rate requirement $R_{\mathrm{req}}$ is calculated. Then, power ${\stackrel{⌣}{p}}_k^c$ is allocated to CU k and, based on this, the data rate $R_d$ for DG d is recalculated.

4. Simulation Results

To assess the efficacy of the joint subcarrier and power allocation algorithm based on improved Exp3 (JSPAA-IExp3) proposed in this paper, its performance is benchmarked against the HSAPP algorithm [23] and the CAFIL algorithm [12] through various simulations. In the following simulation experiments, unless otherwise specified, the relevant simulation parameters are shown in

Table 3. Relevant simulation parameters and values.

Parameters	Value
Cellular cell radius R	200 m
Maximum radius of DGs	20 m
Position of CUs and users of DGs	Uniformly distributed in the semicircle of $\left[0,R\right]$
Number of CUs K	9
Number of DGs D	3
Number of receiving devices M within each DG	3
Number of BS antennas	512
Number of subcarriers C	10
Path loss constant $\alpha$	2.2
Number of multipaths L	3
Carrier center frequency	30 GHz
Maximum power of CUs $p_k^{max}$	20 dBm
Maximum power of DGs $p_d^{max}$	17 dBm
Data rate requirement of CUs $R_{\mathrm{req}}$	1 bps/Hz
SINR	20 dB
Time period T	1000 slots

. We first compare the computational complexity of the different schemes in

Table 4. Comparison of computational complexity of different schemes.

Algorithm	Computational Complexity
JSPAA-IExp3	$O\left(\left|\mathcal{D}\right||{\mathcal{A}}_d^{\mathrm{re}}|T\right)$
HSAPP	$\begin{array}{c}\hfill O({\left(2\left|\mathcal{C}\right|\left(\right|\mathcal{K}|+\left|\mathcal{D}\right|)+\left|\mathcal{K}\right|+\left|\mathcal{K}\right|f\left({\mathcal{D}}_D\right)\right)}^{0.5}\hfill \\ \hfill \left(3\left|\mathcal{C}\right|\left(\right|\mathcal{K}|+\left|\mathcal{D}\right|)+\left|\mathcal{K}\right|+\left|\mathcal{D}\right|f\left({\mathcal{D}}_D\right)\right)\left(\right|\mathcal{K}|+{\left|\mathcal{D}\right|)}^2{\left|\mathcal{C}\right|}^2)\hfill \end{array}$
CAFIL	$O\left(\right|\mathcal{K}|+\left|\mathcal{D}\right|\left(\right|{\mathcal{D}}_d|+1))$

, then present the convergence behavior of the proposed algorithm, and finally investigate the relationship between the total data rate of the DGs and several simulation parameters, including the number of DGs, the number of CUs, the number of subcarriers, and the data rate requirements of the CUs.

Figure 3 illustrates the dynamic evolution of the total multicast data rate for the DGs across successive time slots under our proposed algorithm. The observed trend—initial rapid growth followed by asymptotic convergence—stems from the carefully calibrated exploration–exploitation trade-off governed by parameter ${\alpha }_z$, which systematically reduces random exploration while prioritizing high-reward subcarrier–power combinations as the slot count increases. Convergence here denotes the algorithm’s stabilization toward near-optimal resource configurations. The residual fluctuations arise from two factors: (1) instantaneous variations in near-field spherical wave channel gains due to user mobility, and (2) the adversarial bandit’s inherent responsiveness to interference pattern changes caused by DG reconfigurations.

Figure 4 demonstrates the scalability of the total multicast data rate across varying numbers of DGs in XL-MIMO networks, comparing our proposed algorithm with the HSAPP and CAFIL benchmarks. The monotonic increase in the total data rate for all methods aligns with XL-MIMO’s spatial multiplexing potential, though our approach consistently outperforms the others due to its adversarial multi-arm bandit architecture. This advantage stems from systematically leveraging historical channel information through the improved Exp3 algorithm’s weighted probability updates, enabling the progressive learning of near-optimal subcarrier–power arm pairs (with each arm representing a discrete subcarrier–power pair) across successive slots. Furthermore, the proposed algorithm consistently achieves a high performance by explicitly modeling many-to-many subcarrier reuse conflicts through adversarial rewards (Equation (11)). In contrast, conventional methods independently determine suboptimal joint subcarrier and power allocations per slot, ignoring temporal correlation in 6G’s quasi-stationary near-field channels.

Figure 5 analyzes the impact of CU density on the total multicast data rate of the DGs in XL-MIMO networks, revealing an inverse correlation where increasing the CU count (from 6 to 10 in simulations) reduces the DG rates across all algorithms. This degradation stems from the proportional growth of subcarriers reserved for CU QoS guarantees (constraint ${\mathrm{C}}_1$ in Equation (10)), constraining the DGs’ spectral reuse opportunities. Our proposed JSPAA-IExp3 algorithm and the HSAPP algorithm mitigate this through their “many-to-many” sharing paradigm, achieving higher DG rates than CAFIL’s restrictive “one-to-one” model by enabling dynamic subcarrier multiplexing across overlapping DGs. Notably, at 10 CUs, our method maintains $80.85\%$ of the peak DG rate observed with 6 CUs, outperforming HSAPP and CAFIL, demonstrating robust scalability for ultra-dense 6G deployments where CU-DG coexistence is inevitable.

Figure 6 evaluates the scalability of a DG’s multicast performance with respect to the available subcarriers in XL-MIMO networks, demonstrating that increasing the subcarrier count (from 8 to 12 in simulations) enhances total DG data rates across all algorithms. Our proposed JSPAA-IExp3 algorithm achieves 7.9–22.2% higher rates than those of HSAPP and CAFIL at 12 subcarriers, primarily due to its adversarial bandit mechanism, which dynamically optimizes “many-to-many” subcarrier reuse while suppressing inter-group interference through learned power adaptation.

Figure 7 investigates the trade-off between CU QoS constraints and DG multicast performance by varying the minimum data rate requirements of the CUs. The observed inverse relationship—where DG rates decrease as CU demands escalate—stems from the dual impact of resource scarcity and interference escalation: higher CU rates necessitate both exclusive subcarrier allocations and elevated transmit power, which collectively reduce the resources available for DGs while amplifying cross-tier interference. Our proposed algorithm outperforms HSAPP and CAFIL through its two-stage adaptive mechanism: (1) the coarse cellular pre-allocation phase strictly enforces CU QoS via convex power budgeting (Equation (13)) and (2) the adversarial bandit phase dynamically redistributes residual subcarriers among DGs using interference-weighted reward updates (Equation (18)), effectively decoupling CU-DG resource competition.

Figure 8 demonstrates the DG multicast performance scaling trends of the proposed scheme versus those of the conventional HSAPP and CAFIL approaches as the BS antenna count increases from 256 to 512 in XL-MIMO deployments. The results reveal that our algorithm achieves an 8.1–19.4% higher total data rate for the DGs than the benchmarks at 512 antennas, with the performance gap expanding linearly due to its enhanced spatial resolution under near-field spherical wave propagation.

5. Conclusions

This paper investigates the problem of joint subcarrier and power allocation in a D2D XL-MIMO communication system. The goal of this problem is to optimize the aggregate multicast data rate for all DGs while maintaining a good QoS for the CUs. To fully explore the benefits of D2D communication, a more general “M2M” sharing scenario was considered. To solve the proposed 0–1 mixed-integer nonlinear programming problem, a phased strategy was adopted. The joint subcarrier and power allocation for all DGs and CUs in the original problem was decoupled into separate joint power and subcarrier allocation problems for the DGs and CUs. In phase one, an adversarial MP-MAB was introduced to solve the joint subcarrier and power allocation problem for the DGs, and the improved Exp3 algorithm was used to solve the problem over slots. This algorithm operates as a fully decentralized method capable of converging to the optimal pairing of subcarriers and power allocation. In phase two, the power was reallocated to the CUs based on the assignment results for the DGs. The experimental results demonstrate that the newly developed algorithm attains at least a $10\%$ improvement in aggregated multicast throughput for D2D communications when benchmarked against the state-of-the-art methods documented in recent research.