Cluster Content Caching: A Deep Reinforcement Learning Approach to Improve Energy Efficiency in Cell-Free Massive Multiple-Input Multiple-Output Networks

Abstract

With the explosive growth of micro-video applications, the transmission burden of fronthaul and backhaul links is increasing, and meanwhile, a lot of energy consumption is also generated. For reducing energy consumption and transmission delay burden, we propose a cell-free massive multiple-input multiple-output (CF-mMIMO) system in which the cache on the access point (AP) is used to reduce the load on the link. In this paper, a total energy efficiency (EE) model of a cache-assisted CF-mMIMO system is established. When optimizing EE, forming the co-operation cluster is critical. Therefore, we propose an energy-efficient joint design of content caching, AP clustering, and low-resolution digital-to-analog converter (DAC) in a cache-assisted CF-mMIMO network based on deep reinforcement learning. This scheme can effectively cache content in APs and select the appropriate DAC resolution. Then, taking into account the channel state information and user equipment (UE)’s content request preference, a deep deterministic policy gradient algorithm is used to jointly optimize the cache strategy, AP clustering, and DAC resolution decisions. Simulation results show that the energy efficiency of the proposed scheme is 4% higher than that of other schemes without the resolution optimization and is much higher than that of the only AP clustering without the joint design of content caching and channel quality.

1. Introduction

Due to the rapid development of smart devices such as smart phones, smart watches, smart robots, and drones, mobile data traffic on wireless networks has experienced terrible growth. IDC estimates that by 2023, there will be 48.9 billion connected devices worldwide [1]. Such a large number of devices will not only generate exabytes of data but will also require massive amounts of content, which will create unprecedented challenges in the upcoming communications field. The capacity of a backhaul link has become the bottleneck of a data-intensive network, and it is necessary to find an efficient way to reduce the backhaul link load to meet the rapidly growing demand for mobile communication.

Caching is a well-known technique used to improve the performance of numerous wired networks, such as content-centric networks [2,3,4]. In cellular networks, caching frequently requesting content at the edge of the network can reduce backhaul costs, reduce access latency and power consumption, and increase throughput. In [5], it is proposed to replace the backhaul link by caching on base stations (BSs). By optimizing the cache strategy, it is possible to serve more users within the limits of download time, which significantly increases throughput. In [6], the caching in the BS can lighten the backhaul traffic load. In order to minimize the overall energy consumption attributed to cache and data transmission including inter-BS and BS-to-server communications, Ref. [7] optimizes the allocation of cache sizes for BSs and service gateways. With the goal of reducing the overall energy consumption of the service, the caching strategy is fine-tuned in [8], where the influence of multicast transmission is considered.

At the same time, to cater to the rising need for traffic data, more and more antennas of base stations and smaller and smaller cell radius will inevitably cause more and more inter-cells interference. To tackle this issue, many research efforts have been made to diminish inter-cell interference [9,10]. Two primary approaches exist: massive multiple-input multiple-output (mMIMO) systems [11] and distributed systems [12]. Within mMIMO systems, BS utilizes the spatial multiplexing yielded by an extensive array of antennas. These antennas are coupled with precoding techniques that can substantially and efficiently mitigate both intra-cell and inter-cell interference among user equipment (UEs). Nonetheless, the uniformity of service quality across all terminals is not guaranteed by the system. The terminal near the BS can enjoy better service due to good channel conditions, while the terminal located at the edge of the cell can only get inadequate quality of service. Within a distributed system, multiple BSs or access points (APs) collaborate by exchanging service data and channel state information (CSI), aiming to minimize inter-cell interference. But distributed systems remain centered around individual cells. Multi-cell collaboration essentially extends the coverage of a single cell. Edge effects continue to impact UE positioned at the periphery of the cell. Therefore, the cell-free massive multiple-input multiple-output (CF-mMIMO) technique has been introduced [13,14], which combines the strengths the aforementioned two systems, namely robust interference cancellation and macro diversity gain. Additionally, it has made some enhancements: (1) It shifts from a cell-centric service model to a UE-centric service model, allowing for potential overlap between distinct AP clusters. (2) There exists a substantial number of APs, wide coverage, and APs are closer to the terminal. Thus, it completely eliminates the concept of a cell.

The essence of the CF-mMIMO system is that the mMIMO system moves the AP closer to the terminals through the integration of fronthaul links and a more frequent utilization of the backhaul links. This will cause a sharp increase in the link load in the CF-mMIMO system, which inevitably results in elevated energy consumption. Therefore, traffic congestion in the fronthaul/backhaul link and high transmission energy consumption constitute the bottlenecks that impede the practical implementation of CF-mMIMO systems. The content caching technology proactively stores data in the cache device and directly transmits the data to the terminal during peak hours without the need to obtain data from the central processing unit (CPU) and core network via the fronthaul/backhaul link. Because the requested content is concentrated in a limited number of popular files [15], the cost associated with its caching is diminishing [16], so content caching proves to be a cost-effective and effective technique in lessening the burden on the link. Building upon this notion, a cache-assisted CF-mMIMO system is introduced in [17]. Moreover, we proposed energy-efficient content of a data caching strategy in CF-mMIMO systems in [18], but only research ideas were provided without experimental validation. Therefore, its total energy efficiency (EE) maximization problem is non-deterministic polynomial-hard (NP-hard) and necessitates solutions through inefficient and non-scalable methods. Additionally, researchers have started to consider the joint optimization of user association and caching strategies [19,20,21]. For example, in [19], the high-density satellite-UAV-terrestrial network scenario is considered, and the initial combination optimization problem is effectively solved using game theory and genetic algorithm for clustering and cache placement, respectively. In [20], for a CF-mMIMO-assisted vehicle edge network, a Deep-Q-Network (DQN) algorithm was proposed to optimize the cache decision for improving the network capacity and content delivery performance. Moreover, two deep reinforcement learning (DRL) methods, single-agent reinforcement learning and multi-agent reinforcement learning, were proposed to solve the joint optimization problem of user association and content cache in CF-mMIMO in [21]. However, most existing research focuses on content caching strategies in edge caching, without considering AP clustering strategies.

On the other hand, a substantial quantity of modules of high-resolution analog-to-digital converters (ADCs) and digital-to-analog converters (DACs) generate a lot of power consumption. To avoid this, it is recommended to use low-resolution ADCs (1–3 bits) in CF-mMIMO networks. This trade-off reduces power consumption while sacrificing spectral efficiency (SE). The work of [22] shows that low-resolution ADCs have better EE than high-resolution ADCs in the uplink of CF-mMIMO systems.

Creating a practical model for the total EE of a cache-assisted CF-mMIMO system, one that is both straightforward to calculate and analyze, while also being amenable to effective optimization, poses a significant challenge. To date, little research has been conducted on cache-assisted CF-mMIMO systems with cache assistance, which encourages the development of this study. The primary contributions of this paper can be outlined as follows:

• In this paper, a new total EE model of a cache-assisted CF-mMIMO system is established, which has the following advantages: the introduction of low-resolution DAC can improve EE; UE-centric cache deployment can provide a better user experience; and considering the influence of different resolution converters on EE, it is more suitable for practical use;
• A deep deterministic policy gradient (DDPG) algorithm is proposed to solve the joint optimization problem of content cache, AP clustering, and DAC resolution, and it can find the global optimal decision for maximizing the EE performance in cache-assisted CF-mMIMO networks;
• We compare and discuss the influence of DAC resolutions, the numbers of UEs, and APs on the EE performance. Moreover, the proposed DDPG method is compared with the benchmark methods, such as clustering based on signal-to-interference noise ratio (SINR) and caching strategies based on content popularity. By exploiting the intelligent design, its EE is not only significantly better than the benchmark (BM) methods but also better than the DDPG method based on the joint content cache and AP clustering.

The rest of the paper is organized as follows. In Section 2, we give a model for the cache-assisted CF-mMIMO system. In Section 3, we propose the total EE model of the cache-assisted CF-mMIMO system and formulate the optimization problem. In Section 4, we present an approach based on DRL. The simulation results and discussion are provided in Section 5. Conclusively, we summarize this paper in Section 6.

2. System Model

In this section, the signal model, cache model, and DAC resolution model of the cache-assisted CF-mMIMO network are introduced. For the signal model, it describes the transmitted signal in the cache-assisted CF-mMIMO network’s downlink channel. For the cache model, a content cache mechanism is outlined to enhance the network’s EE. For the low-resolution DAC model, the power consumption generated with different resolutions and the effect on the signal transmission are explained.

2.1. Signal Model

Figure 1 depicts an example topology of a dynamic collaborative cluster serving UE in a cache-assisted CF-mMIMO network. We consider a downlink CF-mMIMO, i.e., a cache-assisted CF-mMIMO system encompassing of M single-antenna AP and K single-antenna UE. Every AP is linked to the CPU via a fronthaul link, while the CPU itself connects to the core network via backhaul links. All APs and UEs are distributed randomly across $S_a$ area. And we only focus on downlink transmissions in this paper.

In time-division duplex (TDD) mode, all APs provide identical time/frequency resources to each terminal. Let the channel linking the m-th AP and the k-th UE be

$$g_{mk}={\left(d_{mk}/d_0\right)}^{-\alpha }{h}_{mk}$$

(1)

where $d_{mk}$ denotes the distance between the m-th AP and the k-th UE, $d_0={\min }_{m,k}{d}_{mk}$ is the reference distance, $\alpha$ represents the path-loss exponent $\left(\alpha \ge 2\right)$, and $h_{mk}\sim \mathcal{C}\mathcal{N}(0,1)$ denotes small-scale fading.

Let ${\Re }_k$ denote the set of APs serving the k-th UE and ${ℂ}_m$ represent the set of UE served by the m-th AP. We make the assumption that each UE is ensured service from no more than $L\left(L<M\right)$ APs (i.e., $\left|{\Re }_k\right|\le L,\forall k$). Therefore, the AP set for all services and the UE set for all services can be represented as $\Re ={\displaystyle {\cup }_{k=1}^K{\Re }_k}$ and $ℂ={\displaystyle {\cup }_{m=1}^M{ℂ}_m}$, respectively. Let $q_k$ be a symbol emitted in the service AP for the k-th UE, where $E[|q_k{|}^2]=1$,$E[q_k]=0,\forall k$ and $E[q_k{q}_l^{\ast }]=0,\forall k\ne l$ (i.e., the symbols of distinct UE are not related). Then, the transmitted signal of the m-th AP can be expressed as [14]

$$x_m={\displaystyle \sum _{k\in {ℂ}_m}\sqrt{p_{mk}}}{\widehat{g}}_{mk}^{*}{q}_k$$

(2)

where $p_{mk}$ signifies the power assigned to the k-th UE at the m-th AP subject to power constraints, $E[|x_m{|}^2]\le P_m$ is constrained by the maximum power $P_m$ transmitted by the m-th AP, and ${\widehat{g}}_{mk}$ denotes the channel estimation $g_{mk}$ at the m-th AP. This paper considers perfect CSI (i.e., ${\widehat{g}}_{mk}=g_{mk},\forall m,k$).

Accordingly, the k-th UE’s received signal can be expressed as [23]

$$\begin{array}{l}{r}_k={\displaystyle \sum _{m\in \Re }{g}_{mk}{x}_m}+w_k\\ \hspace{1em}={\displaystyle \sum _{m\in {\Re }_k}{g}_{mk}{x}_m}+{\displaystyle \sum _{m\in {\Re }_k^c}{g}_{mk}{x}_m}+w_k\\ \hspace{1em}={\displaystyle \sum _{m\in {\Re }_k}{\displaystyle \sum _{k^{\prime }\in {ℂ}_m}\sqrt{p_{m{k}^{\prime }}}{g}_{mk}{\widehat{g}}_{m{k}^{\prime }}^{*}{q}_{k^{\prime }}}}+{\displaystyle \sum _{m\in {\Re }_k^c}{g}_{mk}{x}_m}+w_k\\ \hspace{1em}=\underset{\mathrm{useful}\mathrm{signal}}{\underbrace{{\displaystyle \sum _{m\in {\Re }_k}\sqrt{p_{mk}}{g}_{mk}{\widehat{g}}_{mk}^{*}{q}_k}}}+\underset{\mathrm{interference}\mathrm{plus}\mathrm{noise}}{\underbrace{{\displaystyle \sum _{m\in {\Re }_{k^{\prime }}}{\displaystyle \sum _{k^{\prime }\in ℂ,k^{\prime }\ne k}\sqrt{p_{m{k}^{\prime }}}{g}_{mk}{\widehat{g}}_{m{k}^{\prime }}^{*}{q}_{k^{\prime }}}+w_k}}}\end{array}$$

(3)

where $w_k\sim \mathcal{C}\mathcal{N}(0,{\sigma }_w^2)$ signifies the noise at the k-th UE, and ${\Re }_k^c=\Re \backslash {\Re }_k$ is the other AP set that does not serve the k-th UE.

2.2. Caching Model

We consider a limited file library $CF=\{c{f}_1,c{f}_2,\cdots ,c{f}_F\}$ with F content files. Let $C{F}_m\subset CF$ be the set of content files cached in the m-th AP. Additionally, we make the assumption that each AP has the capacity to cache a maximum of $N\left(N<F\right)$ files, denoted as $\left|C{F}_m\right|\le N,\forall m$. Each UE makes content file requests independently or abandons the request. The content file for the k-th UE request is represented by $c{f}_k\in CF$, where $c{f}_k$ is determined by the content preference vector for the k-th UE (arranged in descending order of preference for all content files) and the distribution of content popularity (specified with the Zipf distribution). To be more specific, in the content preference vector of the k-th UE, the probability that $c{f}_k$ equals the content file of the i-th rank is $i^{-\beta }/{\displaystyle {\sum }_{j=1}^F{j}^{-\beta }}$, where $\beta$ is the Zipf factor; Usually, set to $\beta =0.5,1,2$. Each UE possesses a different, independent, and time-invariant content preference vector.

We use $H_{\mathrm{m}k}$ to define the event that the content file requested by the k-th UE is cached on its m-th service AP, i.e., $c{f}_k\in C{F}_m,m\in {\Re }_k$. Therefore, the matching event of the k-th UE $H_k$ indicates that the file requested by the k-th UE is cached across all APs serving the k-th UE, i.e., $c{f}_k\in C{F}_m,\forall m\in {\Re }_k$. In case of a miss, there exist certain $m\in {\Re }_k$ APs that do not cache the file of the k-th UE, i.e., $c{f}_k\notin C{F}_m,\exists m\in {\Re }_k$. In such scenarios, these APs will necessitate requesting the content file $c{f}_k$ from the CPU/core network for joint AP transfer. The network’s hit ratio is denoted as $H={\displaystyle {\sum }_{k\in ℂ}{1}_{H_k}}/|ℂ|$, where $1_{H_k}$ signifies the indicator function, and it is set to be 1 if the $H_k$ event occurs; otherwise, it is set to 0.

2.3. Low-Resolution DAC Model

We adopt a low-resolution DAC with a binary-weighted current-oriented topology, whose power consumption is composed of both static and dynamic components. The power consumption of a DAC module with a resolution of b can be given as [24,25]

$$P_{DAC}(b,F_s)=1.5\times {10}^{-5}\cdot 2^b+4.5\times {10}^{-12}\cdot b\cdot F_s$$

(4)

where $F_s$ is the sampling frequency.

Each AP’s antenna is connected to a low-resolution DAC, and the resulting signal has $\alpha \in [0,1]$ linear gain. Therefore, the transmitting signal of the m-th AP given in (2) is now modified as

$$x_m={\alpha }_m{\displaystyle \sum _{k\in {ℂ}_m}\sqrt{p_{mk}}}{\widehat{g}}_{mk}^{*}{q}_k$$

(5)

where ${\alpha }_m$ represents the linear gain of the m-th AP, and its expression is [22,26]

$${\alpha }_m=\{\begin{array}{r}0.6366\hspace{1em}\hspace{1em}\hspace{0.17em},b_m=1\\ 0.8825\hspace{1em}\hspace{1em}\hspace{0.17em},b_m=2\\ 1-(\pi \sqrt{3}/2)\cdot 2^{{}^{-2b_m}},b_m\ge 3\end{array}$$

(6)

The received signal of k-th terminal provided by (3) is rewritten as

$$r_k=\underset{\mathrm{useful}\mathrm{signal}}{\underbrace{{\displaystyle \sum _{m\in {\Re }_k}{\alpha }_m\sqrt{p_{mk}}{g}_{mk}{\widehat{g}}_{mk}^{*}{q}_k}}}+\underset{\mathrm{interference}\mathrm{plus}\mathrm{noise}}{\underbrace{{\displaystyle \sum _{m\in {\Re }_{k^{\prime }}}{\alpha }_m{\displaystyle \sum _{k^{\prime }\in ℂ,k^{\prime }\ne k}\sqrt{p_{m{k}^{\prime }}}{g}_{mk}{\widehat{g}}_{m{k}^{\prime }}^{*}{q}_{k^{\prime }}}+w_k}}}$$

(7)

3. The EE Model and Problem Formulation

3.1. The System Sum Rate

According to the Shannon theory, the achievable rate of the k-th UE can be expressed as

$$R_k=B{\log }_2\left(1+\frac{{\left|{\displaystyle \sum _{m\in {\Re }_k}{\alpha }_m\sqrt{p_{mk}}{g}_{mk}{\widehat{g}}_{mk}^{*}}\right|}^2}{{\displaystyle \sum _{k^{\prime }\in ℂ,k^{\prime }\ne k}{\left|{\displaystyle \sum _{m\in \Re }{\alpha }_m}\sqrt{p_{m{k}^{\prime }}}{g}_{mk}{\widehat{g}}_{m{k}^{\prime }}^{*}\right|}^2}+{\left|w_k\right|}^2}\right)$$

(8)

where B is the bandwidth. Therefore, the overall achievable rate of the considered cache-assisted CF-mMIMO network is given using

$$R_{sum}={\displaystyle \sum _{k\in ℂ}{R}_k}$$

(9)

3.2. Power Consumption

The overall power consumption of the network consists of four parts: (1) the transmission power of all service APs; (2) power consumption of DAC in all service APs; (3) the power required by the AP to recover the lost content file from the CPU; and (4) the power needed by the CPU to recover the lost content from the core network.

For (1), the total transmitted power of all service APs is represented by ${\displaystyle {\sum }_{m\in \Re }{P}_m}$. For (2), the sum of DAC power consumption in all service APs is given using ${\displaystyle {\sum }_{m\in \Re }{P}_{DAC}^m}$, where $P_{DAC}^m$ indicates the power consumption of the DAC module with $b_m$ resolution selected using the m-th AP.

For (3) and (4) in cache-assisted CF-mMIMO systems, all APs within a cluster must concurrently transmit identical content to the terminal. Cache deployment results in three scenarios are illustrated in Figure 2: (a) Every AP within the cluster has deployed the required content. (b) Only some APs have deployed the required data. c) None of the APs deployed the required content. In Scenario (a), no content is conveyed through either the fronthaul or backhaul link. In Scenario (b), certain APs are required to transmit content via the fronthaul link. In Scenario (c), all content is transmitted to the AP via the backhaul link of the core network and the fronthaul link of the CPU.

The fronthaul link is utilized for content transmission between the AP and the CPU, its power consumption is proportional to the cumulative SE sum, and its expression is [27]

$$P_{bh,m}=E_{bh}{\displaystyle \sum _{k\in {ℂ}_m}{R}_k}$$

(10)

where $E_{bh}$ indicates the energy consumed for transmitting 1 Mbit of data over the fronthaul link.

The m-th AP is used to transmit data $q_1,q_2,\cdots q_K$ via a fronthaul/backhaul link between the CPU and core network. Therefore, the fronthaul/backhaul power consumption depends on the SE, ${\mathrm{SE}}_1,{\mathrm{SE}}_2,\cdots {\mathrm{SE}}_K$. If the m-th AP serves only specific UEs, it merely transmits data related to these UEs. Therefore, the power consumption for the fronthaul/backhaul is contingent solely on the SE of these UEs. As shown in Figure 2, the cache power consumption is calculated with the user as the center, so the fronthaul power consumption of the k-th cluster can be represented as

$$P_{bh,k}=E_{bh}{R}_k$$

(11)

Similarly, the backhaul power generated by the k-th cluster’s backhaul link for transferring data between the core network and the CPU can be expressed as

$$P_{bb,k}=E_{bb}{R}_k$$

(12)

where $E_{bb}$ indicates the energy consumed for transmitting 1 Mbit of data over a backlink. Therefore, the power uploaded via the AP to the CPU in the k-th cluster can be represented as

$$P_{bh,k}^{up}=\{\begin{array}{l}{P}_{bh,k},\hspace{1em}0<{\displaystyle \sum _{m\in {\Re }_k}\left|H_{mk}^{miss}\right|}/\left|{\Re }_k\right|<1\\ \hspace{1em}\hspace{0.17em}0,\hspace{1em}\mathrm{others}\end{array}$$

(13)

where $H_{mk}^{miss}=\left|1-H_{mk}\right|$ represents the event that the UE content request provided by the m-th AP is not cached on the m-th AP. The value is 1 if the event $H_{mk}^{miss}$ occurs, but it is 0 otherwise.

The energy consumption associated with the content requested by the AP from the CPU in the k-th cluster can be expressed as

$$P_{bh,k}^{down}=P_{bh,k}{\displaystyle \sum _{m\in {\Re }_k}\left|H_{mk}^{miss}\right|}$$

(14)

The backhaul power generated by the k-th cluster requesting content from the core network can be denoted as

$$P_{bb,k}^{down}=P_{bb,k}\lfloor {\displaystyle \sum _{m\in {\Re }_k}\left|H_{mk}^{miss}\right|}/\left|{\Re }_k\right|\rfloor$$

(15)

Therefore, the fronthaul/backhaul power consumption of the k-th cluster can be expressed as

$$P_{B,k}=P_{{}_{bh,k}}^{up}+P_{{}_{bh,k}}^{down}+P_{{}_{bb,k}}^{down}$$

(16)

So, the overall energy consumption can be expressed as

$$P_{total}={\displaystyle \sum _{m\in \Re }\left(P_m+P_{DAC}^m\right)}+{\displaystyle \sum _{k=1}^K{P}_{B,k}}$$

(17)

3.3. Problem Formulation

Our aim is to find a strategy that determines the AP clustering with ${\Re }_1,{\Re }_2,\cdots ,{\Re }_K$, as well as the AP’s content cache $C{F}_1,C{F}_2,\cdots ,C{F}_M$ and its DAC resolution $b_1,b_2,\cdots ,b_M$, in order to maximize the system’s EE. The optimization problem can be expressed as

$$\begin{array}{l}\max \hspace{1em}\frac{R_{sum}}{P_{total}}\\ s.t.\hspace{1em}\hspace{0.17em}\left(\mathrm{C}1\right):\left|{\Re }_k\right|\le M,\forall k\in ℂ,\\ \hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\left(\mathrm{C}2\right):\left|C{F}_m\right|\le L,\forall m\in \Re ,\\ \hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\left(\mathrm{C}3\right):b_m\in N^{+},\forall m\in \Re .\end{array}$$

(18)

where constraint (C1) indicates that the count of APs within the AP cluster of each UE cannot exceed the maximum number of connections M. Constraint (C2) requires that the amount of content cached on each AP must not exceed its maximum capacity number L. Constraint (C3) means that the resolution of each DAC is a positive integer.

To maximize the EE performance, the design for trade-offs is needed. First, AP clusters based on channel quality and high-resolution DAC can select better channels to get the best SE. In contrast, an AP cluster based entirely on cached content and low-resolution DAC can avoid the energy consumption of the fronthaul/backhaul link, reducing the energy consumption of the DAC module. In addition, in large networks, solving this issue is complicated due to the large number of APs and UEs. To solve this problem, we developed deep reinforcement learning-based content caching, AP clustering, and DAC resolution co-selection strategies, which will be elaborated upon in the Section 4.

4. Deep Reinforcement Learning Method

In this section, we will describe how the DDPG algorithm solves the combined predicament of AP clustering, caching, and selecting DAC resolution. Three basic components (action, state, and reward) are defined in reinforcement learning (RL) problems.

4.1. Action, State, and Reward

In slot t, action $a_t$ encompasses the processes of clustering, caching, and selecting resolution. Let $a_{mk,t}\in \{0,1\}$, $a_{mcf,t}\in \{0,1\}$, and $a_{mb,t}\in \{0,1\}$ represent the status of m-th AP and k-th UE service, the cf file cache, and the b bit resolution switch, respectively, where “1” indicates that the service or cache is successful or enabled and “0” indicates that the service or cache is not served, there is no cache, or it is disabled. So, action $a_t$ can be defined as

$$a_t\triangleq \left\{a_t^{cl},a_t^{ca},a_t^{res}\right\}$$

(19)

The sets $a_t^{cl}=\{a_{mk,t}:m\in M,k\in K\}$, $a_t^{ca}=\{a_{mcf,t}:m\in M,cf\in CF\}$, and $a_t^{res}=\{a_{mb,t}:m\in M,b\in N^{+}\}$ contain aggregate results representing the t-th time slot for clustering, caching, and selecting resolution, respectively.

Similarly, the action a_t uniquely determines the sets $C{F}_m$, $b_m$, ${\Re }_k$ and ${ℂ}_m$, i.e., $C{F}_m=\{cf:a_{mcf,t}=1,cf\in CF\}$, $b_m=\{b:a_{mb,t}=1,b\in N^{+}\}$, ${\Re }_k=\{m:a_{mk,t}=1,m\in M\}$, and ${ℂ}_m=\{k:a_{mk,t}=1,k\in K\}$.

The state considered in RL should be the set of information that the CPU can collect to compute the reward. In this article, the state of the t-th slot is characterized as the collection of channel gain $G_t=\{g_{mk,t}:m\in M,k\in K\}$, the action of the preceding time slot, and the historical record of file requests for each UE. Define the history set of user requests as $e_t=\{e_{kcf,t}:k\in K,cf\in CF\}$, where $e_{kcf,t}={\displaystyle {\sum }_{t^{\prime }=1}^t{1}_{c{f}_{k,t^{\prime }=cf}}}$ is the cf file download from the k-th UE request as of time t. So, the state can be denoted as

$$s_t\triangleq \left\{G_t,a_{t-1},e_t\right\}$$

(20)

According to the objection function of the optimization problem (18), the reward function of the t-th slot is defined as

$$r\left(s_t,a_t\right)\triangleq \frac{R_{sum,t}}{P_{total,t}}$$

(21)

where $R_{sum,t}$ and $P_{total,t}$ are given in (9) and (17), correspondingly, and the extra subscript t is given to emphasize the dynamic behavior. It is worth noting that the total achievable rate $R_{sum,t}$ is contingent upon the channel conditions $G_t$, the clustering outcome $a_t^{cl}$, and the result $a_t^{res}$ of the selection resolution, while the overall power $P_{total,t}$ is contingent upon the caching result $a_t^{ca}$ and the result $a_t^{res}$ of the selection resolution.

4.2. Deep Deterministic Policy Gradient Approach

DDPG algorithm utilizes an actor–critic network architecture. Moreover, each network is accompanied by its respective target network, resulting in a total of four networks within the DDPG algorithm, namely, the actor network $\mu (\cdot |{\theta }^{\mu })$, critic network $Q(\cdot |{\theta }^Q)$, target actor network ${\mu }^{\prime }(\cdot |{\theta }^{{\mu }^{\prime }})$, and target critic network $Q^{\prime }(\cdot |{\theta }^{Q^{\prime }})$. Each network updates according to its own update rules, maximizing cumulative expected returns. Figure 3 gives the schematic diagram of the DDPG algorithm.

The DDPG algorithm is well-suited for multi-task learning, aligning with the objectives of this paper. This algorithm enhances training stability by adopting a deterministic strategy, which means it directly outputs a specific action value instead of a probability distribution. The algorithm is trained using an experience replay buffer to store past experiences and then to randomly sample from it. This approach breaks the data correlations and ensures that the data conform to an independent distribution, thereby reducing parameter update variance and enhancing convergence speed. Additionally, experiences can be reused, resulting in high data utilization. DDPG leverages neural networks to represent policies (actor) and value functions (critic), making it suitable for high-dimensional state spaces and capable of learning from vast amounts of perceptual data. In comparison to the widely used DQN, DDPG is particularly apt for continuous action spaces. Furthermore, employing actor networks can improve training efficiency, and having more target actor networks and target critic networks helps prevent potential overestimation issues present in DQN.

Algorithm 1 primarily revises the parameters of the actor network and critic network. The actor network adjusts the weight ${\theta }^{\mu }$ by aiming to maximize the cumulative anticipated reward. The critic network adjusts the weight ${\theta }^Q$ by seeking to minimize the discrepancy between the evaluation value and the target value. Regarding the update process of the target network, a soft update method is adopted, which can also be called exponential average motion. That is, the learning rate (or momentum) $\tau$ is introduced, and the weighted average of the previous target network parameters and the current corresponding network parameters are subsequently applied to update the target network. Algorithm 1 summarizes the whole DDPG algorithm process.

Algorithm 1 DDPG Algorithm Procedure

$1:\mathrm{Actor}–\mathrm{critic}\mathrm{network}\mathrm{parameter}{\theta }^{\mu }$$\mathrm{and}{\theta }^Q$ initialization

$2:\mathrm{Set}\mathrm{the}\mathrm{same}\mathrm{parameters}{\theta }^{{\mu }^{\prime }}$$\mathrm{and}{\theta }^{Q^{\prime }}$ in the target network

3: for plot = 1 to Plot do

4:   for timeslot = 1 to T do

$5:\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{Generate}\mathrm{action}{a}_t$$\mathrm{through}\mathrm{the}\mathrm{actor}\mathrm{network}\mu (s_t|{\theta }^{\mu })$

$6:\hspace{1em}\hspace{1em}\hspace{1em} \mathrm{Get}\mathrm{rewards}r\left(s_t,a_t\right)$$\mathrm{and}\mathrm{next}\mathrm{status}{s}_{t+1}$$\mathrm{according}\mathrm{to}\mathrm{the}\mathrm{action}{a}_t$

$7:\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{Get}\mathrm{the}\mathrm{target}\mathrm{value}q\mathrm{through}\mathrm{the}\mathrm{critic}\mathrm{network}Q(s_t,a_t|{\theta }^Q)$

$8:\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{Use}\mathrm{the}\mathrm{target}\mathrm{network}{Q}^{\prime }(s_{t+1},{a^{\prime }}_{t+1}|{\theta }^{Q^{\prime }})\mathrm{to}\mathrm{get}\mathrm{the}\mathrm{separate}\mathrm{target}\mathrm{value}y$

$9:\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{The}\mathrm{gradient}\mathrm{is}\mathrm{determined}\mathrm{by}\mathrm{the}\mathrm{target}\mathrm{value}q\mathrm{of}\mathrm{the}\mathrm{actor}–\mathrm{critic}\mathrm{network}\mathrm{and}\mathrm{the}\mathrm{target}\mathrm{network}\mathrm{target}\mathrm{value}y$

10: $\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{Update}\mathrm{parameters}{\theta }^{\mu }$$\mathrm{and}{\theta }^Q$ in the network of actors and critics according to the gradient

11: $\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{Update}\mathrm{parameters}{\theta }^{{\mu }^{\prime }}$$\mathrm{and}{\theta }^{Q^{\prime }}$ in the target network according to the$\mathrm{parameters}{\theta }^{\mu }\mathrm{and}{\theta }^Q\mathrm{in}\mathrm{the}\mathrm{actor}\mathrm{and}\mathrm{critic}\mathrm{network}\mathrm{and}\mathrm{the}\mathrm{learning}\mathrm{rate}\tau$

12:   end for

13:  end for

4.3. Computational Complexity

In the DDPG algorithm, the input dimension of the neural network is denoted as $Input\triangleq M(3K+N+1)+KN$, the output dimension is denoted as $Output\triangleq M(K+N+b)$, and the number of model parameters is denoted as $Number\triangleq 5Input(Input+1)+9Output(Output+1)+10Iuput\ast Output$, determined by the neural network’s layer count and layer size. The experience pool’s size is $Batch=128$, and it holds states, actions, rewards, and next states, resulting in a complexity of $O(Batch\ast (2K(M+N+b)+K(M+N+1)+1)+Number)$. The decision-making process for actions has a time complexity of $O(K+2M)$, thus leading to the complexity of $O(timeslot\ast (Number+K+2M))$, where $timeslot$ stands for the number of training iterations.

5. Simulation Results

5.1. Simulation Settings

In this section, we conduct a comparison and analysis of (1) the EE performance of the proposed RL method with three different BM strategies (called BM1, BM2, and BM3), (2) the convergence behavior of DDPG algorithm, (3) the effect of DAC resolutions on the EE, (4) the impact of the number of UE-associated AP on the EE, and (5) the influence of UE and AP quantity on the EE. The BM strategies are given as follows:

• BM1: clustering policy based on the SINR ($l\le L$ APs to which the k-th UE is connected is the l with the highest SINR) and caching policy based on local popularity (in the UE served by the m-th AP, the most popular N files are cached on the m-th AP);
• BM2: clustering strategy based on SINR (same as BM1) and caching strategy based on network popularity (caching the N most popular files across all UEs in all APs);
• BM3: cache-based clustering strategy (each UE is connected to $l\le L$ APs, and its cache is the content request that best matches each UE in the previous slot) and network-based caching strategy (same as BM2).

The computational complexity of BM strategies is all equal to $O(K(M+N+1))$, with BM2 having the minimum time complexity of $O((M{\log }_{2}M)\ast (KN{\log }_{2}N))$, BM3 following with $O((KM{\log }_{2}M)\ast (KN{\log }_{2}N))$, and BM1 having the maximum complexity of $O((M{\log }_{2}M)\ast (MKN{\log }_{2}N))$. Although the complexity of the BM strategy is lower than that of DDPG algorithm, the optimization effect of the DDPG algorithm is much better than that of the BM strategy.

We contemplate a situation where APs and UEs are randomly distributed within the region of $S_a=1\mathrm{k}{\mathrm{m}}^2$, one AP is located at the reference coordinate $(0,0)$. Both the positions of UEs and APs remain constant during the training phase. We set the number of APs and UEs to be $M=10$ and $K=5$, correspondingly, the cache size to $\left|C{F}_m\right|\le 2$ for each AP, the number of files to $\left|CF\right|=10$, and the DAC resolution to $b_m\le 5$. Refer to [14,28,29,30,31,32] for other system settings and parameters, which are summarized in

Table 1. The simulation parameters.

Parameters	Value
Bandwidth B	20 MHz
$\mathrm{Maximum}\mathrm{DL}\mathrm{transmit}\mathrm{power}{P}_m$	1000 mW
$\mathrm{Energy}\mathrm{consumption}\mathrm{of}\mathrm{fronthaul}\mathrm{link}{E}_{bh}$	$0.25\times {10}^{-3}$ Joule/Mbit
$\mathrm{Energy}\mathrm{consumption}\mathrm{of}\mathrm{backhaul}\mathrm{link}{E}_{bb}$	$15E_{bh}$
$\mathrm{Thermal}\mathrm{noise}\mathrm{power}\mathrm{per}\mathrm{UE}{\sigma }_w^2$	$7.457\times {10}^{-13}$ W
Path-loss exponent α	2
Zipf distribution factor β	1

.

5.2. Numerical Results Analysis

Figure 4 shows the convergence of RL+DAC and RL in Algorithm 1, where the EE values versus training episodes are demonstrated. The diagram is trained 1000 times and then fused together. When the BM strategy is compared at the 10th episode of training, it can be seen that the EE of RL+DAC and RL proposed after the 26th episode of training is better than that of other BM strategies. The EE of RL+DAC also completely outperformed the RL algorithm after about the 150th episode of training. Note that in Algorithm 1, RL+DAC assumes that each AP employs a distinct ADC resolution, which sacrifices some computational time in exchange for the improved performance. In contrast, RL employs the same ADC resolution for all APs, thereby reducing the algorithm’s complexity.

Figure 5 illustrates the impact on total EE of changes in the relative positions of APs and UEs over time. It can be easily observed that the total EE of the RL+DAC and RL algorithms are always better than that of other BM strategies. In BM schemes, we find that when each UE is attended to by a single AP at moments 0, 3, 4, 7, 8, 10, and 11, their total EE values are higher than others. When each UE is attended to by three AP at moments 1, 5, 6, and 9, the total EE performance is the highest. What this means is that one UE does not choose more AP services to get a better EE performance. Furthermore, the increasing of the number of UEs will result in a greater number of UEs being served by the APs. Consequently, this necessitates APs to make trade-offs when selecting DAC resolution with the RL+DAC algorithm, significantly diminishing the SE improvement for UEs. When there is an abundance of UEs, this effect becomes nearly equivalent to the average SE achieved in the case of employing the same low resolution at each AP.

Figure 6 depicts the correlation between the number of UEs and the mean SE, where the number of APs is $M=10$. As illustrated in Figure 6, the mean SE of UEs diminishes as the quantity of UEs increases and then tends to be stable. This phenomenon arises due to the escalation in the quantity of UEs, leading to a gradual intensification of inter-UE interference. Ultimately, the average SE will become stable. Furthermore, the upsurge in the number of UEs will result in a greater number of UEs being served by the APs. Consequently, this necessitates APs to make trade-offs when selecting DAC resolution within the RL+DAC algorithm, significantly diminishing the SE improvement for UEs. When there is an abundance of UEs, this effect becomes nearly equivalent to the average SE achieved when each AP in the RL algorithm utilizes the same low resolution.

Figure 7 explores the influence of the number of UEs on the total EE, where the number of APs is $M=10$. As observed in Figure 7, the overall trend of the total EE decreases as the quantity of UEs increases and then tends to stabilize. This is because the growth of the number of UEs in the early stage is approximately proportional to the energy consumed by the system, and the existence of inter-UE interference will slow the growth of its sum achievable rate, so its total EE continues to decline. When the number of UEs is large, all APs are already serving UE, and augmenting the number of UEs will not lead to an elevation in the power consumption of AP activities, resulting in a smaller increase in total power consumption, so the total EE will tend to balance. It also indirectly validates the result of Figure 6: the increase in the number of UE does not always guarantee better overall system performance. In other words, the higher the number of UE, the interference between UEs will be particularly significant. Note: the total EE of $K=3$ UEs in Figure 7 is lower than the total EE of $K=4$. This is due to the different location of AP and UE, which will lead to different channel conditions, so that the total EE will produce a certain range fluctuation when the quantity of UE is determined. When the quantity is smaller, the fluctuation due to the different effects of the location will be greater.

Figure 8 shows the correlation between the quantity of APs and the sum achievable rate, where the number of UEs is set as $K=5$. It is readily noticeable that the sum achievable rate firstly increases with the number of APs and then tends to be stable. This is because when the number of APs is small, the UE selects the APs with better channel conditions, so that the rate can be increased. Nevertheless, in scenarios where the number of APs is large, the augmentation for the number of APs brings about a gradual intensification of interference between APs. Consequently, when the number of APs is already relatively high, further increasing the number of APs will not lead to an increase in the sum achievable rate; in fact, it might even decrease it. Moreover, as depicted in Figure 8, the curves for BM1 $(l=3)$ and BM2 $(l=3)$ overlap due to their shared clustering policies, distinct caching strategies, and the fact that the sum achievable rate is solely contingent on SINR and not influenced by caching.

Figure 9 shows the impact of the total quantity of AP on the total EE, where the quantity of UE is $K=5$. The simulation diagram also indirectly verifies the result of Figure 8, i.e., the more AP, the overall system performance is not necessarily better. At the same time, it illustrated from Figure 9 that when the quantity of APs is 16, the total EE of the system is the highest. Because as the quantity of APs increases, so does their power consumption, and when their sum achievable rate increases slowly, their total EE will begin to decrease. In addition, note that the sum achievable rate and total EE of 20 for the number of AP in Figure 8 and Figure 9 do not strictly follow the trend. This is caused by fluctuations due to the randomness of the positions of APs and UEs.

In Figure 10, the impact of low-resolution DAC on the total EE is demonstrated, and it can be observed that when $b\ge 6$, the total EE decreases as the resolution $b$ increases. This means that resolution $b$ will achieve a better total EE performance in the interval $[1,5]$, and the RL+DAC resolution in the figure is $b_m\le 5,\forall m\in \Re$, which has the best total EE. Therefore, this also validates the wisdom of limiting the resolution range to $b\le 5$ in our RL+DAC algorithm design. In addition, with the increasing resolution $b$, its total EE decreases faster and faster because in Formula (4), part of the DAC module’s power consumption increases exponentially with the increase in resolution $b$, while in Formulas (6) and (8), with the resolution $b>5$, it becomes evident that the sum achievable rate tends to be stable.

6. Conclusions

In this paper, an innovative and practical total EE model of a cache-assisted CF-mMIMO system is established. To maximize the total EE, an energy-efficient joint design of content cache, AP clustering, and low-resolution DAC is carried out, and then, a DRL algorithm (i.e., DDPG method) is proposed. Numerical results show that the total EE of the RL+DAC strategy considering DAC resolution is generally 4% higher than that of the RL strategy, and the total EE of these strategies is much higher than those of the BM strategies. In addition, it can be expected that for multi-antenna APs, the number of DAC modules increases linearly with the increase in the quantity of antennas, so the total EE of our proposed RL+DAC strategy will be much higher than that of the RL strategy.