Data-Driven Joint Beam Selection and Power Allocation for Multiple Target Tracking

Abstract

For the problem of joint beam selection and power allocation (JBSPA) for multiple target tracking (MTT), existing works tend to allocate resources only considering the MTT performance at the current tracking time instant. However, in this way, it cannot guarantee the long-term MTT performance in the future. If the JBSPA not only considers the tracking performance at the current tracking time instant but also at the future tracking time instant, the allocation results are theoretically able to enhance the long-term tracking performance and the robustness of tracking. Motivated by this, the JBSPA is formulated as a model-free Markov decision process (MDP) problem, and solved with a data-driven method in this article, i.e., deep reinforcement learning (DRL). With DRL, the optimal policy is given by learning from the massive interacting data of the DRL agent and environment. In addition, in order to ensure the information prediction performance of target state in maneuvering target scenarios, a data-driven method is developed based on Long-short term memory (LSTM) incorporating the Gaussian mixture model (GMM), which is called LSTM-GMM for short. This method can realize the state prediction by learning the regularity of nonlinear state transitions of maneuvering targets, where the GMM is used to describe the target motion uncertainty in LSTM. Simulation results have shown the effectiveness of the proposed method.

1. Introduction

Multiple target tracking (MTT) is an important research topic in the military field, and also a difficult problem at present [1,2,3]. Multiple-input multiple-output (MIMO) [4] radar is an extension of traditional phased array radar, and it is able to transmit signals in all directions and multi-beams at the same time. This operating mode is called simultaneous multi-beam mode [5]. Technically speaking, in the simultaneous multi-beam mode, a single collocated MIMO radar can track multiple targets simultaneously by controlling different beams tracking different targets independently. In this case, the complex data association process of tracking can be avoided since the spatial diversity of multiple targets can be used to distinguish the echoes of different targets, and thus, the multi-target tracking problem can be divided into multiple single-target tracking problems.

In practical applications, we have to consider two limiting factors inherent to collocated MIMO radar [6]. (1) The maximum number of beams that the system can generate is limited at each time. In theory, if the number of MIMO radar transmitters is N, the system can only generate M ($M<N$) orthogonal beams at most at each time. (2) The total transmitted power of simultaneous multiple beams is limited by the hardware. In real applications, it is necessary to limit the total transmitted power of multiple beams to ensure that the total transmission power of the system does not exceed the tolerance of the hardware. With the traditional transmitting mode, the number of beams is always set to a constant, and the transmitting power is uniformly allocated to the beams. Although it is easy to implement in engineering, the utilization efficiency of the resource is relatively low. In terms of the kinds of resources, the existing resource allocation methods can be roughly divided into two groups: one is based on the transmitting parameters, the other is based on the system composition structure. In this paper, the discussed problem of power and beam resource allocation belongs to the first one. The resource allocation based on the system composition structure mostly exists in multiple radar systems (MRS).

In [7], the authors used the covariance matrix of the tracking filter as the evaluation of tracking performance, and established and solved an optimization cost function of transmitted power. The authors of [8] proposed a cognitive MTT method based on prior information, and designed a cognitive target tracking framework. With this method, the radar transmit power parameters can be adaptively adjusted at each tracking time based on the information fed back by the tracker. In [9], an MRS resource allocation method is proposed for MTT based on quality of service constrained (QoSC). This method cannot only illuminate multiple targets in groups, but also optimize the transmitted power of each radar. The authors of [10,11,12,13,14] also studied the optimization-based resource allocation method with different resources limitation in different target–radar scenarios. Their similarity is to take the localization Cramér–Rao lower bound (CRLB) or the tracking Bayesian Cramér–Rao lower bound (BCRLB) [13] as the performance evaluation in the cost function. However, the used CRLBs and BCRLBs are both derived with an assumption of constant velocity (CV) model or constant acceleration (CA) model, we thus conclude that these kinds of methods are model-based methods. These model-based methods can achieve excellent results on the premise that the target’s motion is conform to the model assumption. However, in the case of maneuvering targets (the targets undergoing non-uniform-linear motions with certain uncertainty and randomness), they would not work well because the inevitable model mismatch causes a lower resource utilization, a larger prediction error of the target state, and an unreliable BCRLB. To overcome this, in [15], the resource problem was first formulated as a Markov decision process (MDP) and solved by a data-driven method, where recurrent neural network (RNN) [16,17] was used to learn a prediction model for BCRLB and target’s state, and deep-reinforcement-learning (DRL) [18,19] was adopted for solving the optimal power allocation solution. This method works under the condition that the number of generated beams is equal to the number of targets. However, in practice, radar system always have to face the challenge that the maximum beam number is far less than the number of targets. On this basis, the authors further propose a constrained-DRL-based MRS resource allocation method for meeting the given accuracy of single target tracking [20]. It is noticed that, in [15,20], the output of RNN is characterized as a single Gaussian distribution. In this way, an accurate predictive information can be obtained as the target is maneuvering with low uncertainty. Based on above studies, in this paper, we will further discuss how to jointly schedule the power and beam resources of a radar system to enhance the overall performance of MTT in maneuvering target scenarios.

In this paper, a novel data-driven joint beam selection and power allocation (JBSPA) method is introduced. The main contributions of this paper can be summarized as follows:

(1)

A JBSPA method for MTT is developed based on deep learning, which aims to maximize the performance of MTT by adjusting power and beam allocation within the limited resources at each illumination. Through mathematical optimization modeling and analysis, the problem of JBSPA is essentially a non-convex and nonlinear problem, and the conventional optimization method is thus not suitable. To address this problem, we characterize the original optimization cost function as a model-free MDP problem by designing and introducing a series of variable parameters of MDP, including state, action and reward. For a discrete MDP, a common way is dynamic programming (DP). However, our problem is a continuous controlling problem over the action and state space, and thus, DP-based methods are not available. Since the emerging DRL is a good model-free MDP [21], it will be taken to solve MDP of JBSPA (DRL-JBSPA) in this paper.

(2)

Considering the possible existence of a model-mismatch problem in maneuvering target tracking, the long short-term memory (LSTM) [17] incorporating the Gaussian mixture model (GMM), say, LSTM-GMM, is used to derive prior information representation of targets for resource allocation. The biggest change exists in the output layer of LSTM; there, two different GMMs are utilized to describe the uncertainty of state transition of the maneuvering target and radar cross-section (RCS), respectively. In this way, at each tracking interval, the LSTM-GMM will not output a determined predicted extended target state, but multiple Gaussian distributions. Then, we select one of them randomly according to mixed coefficients, to obtain the predicted target state and RCS, and the Bayesian information matrix (BIM) of prior information at the next tracking time instant.

(3)

Considering that the JBSPA is a continuous control problem over state and action spaces, in order to improve the computational efficiency, this work adopts a deterministic policy function in calculation of Q function, and deduces the deterministic policy gradient [22,23] by deriving the Q function with respect to policy parameters. Then, we apply an actor network and a critic network to enhance the stability of the training. A modified actor network is designed to improve the applicability of the original actor-critic frame to JBSPA, where the output transmit parameters (i.e., the number of generated beams by the radar, the pointing of beams, and the power of each beam) are realized by taking three different output layers into the action network. Due to the usage of a trained network and avoiding optimization solving at each tracking time, the requirement of real time in resource allocation can be guaranteed in MTT.

The outline of the paper is as follows. In Section 2, a brief review of DRL is introduced. In Section 3, the resource allocation problem is described. In Section 3, the resource allocation problem is abstracted into a mathematical cost function. In Section 4, a deep-learning-based JBSPA method is detailed. In Section 5, several numerical experiment results are provided to verify the effectiveness of the proposed method. Some problems and future works are discussed in Section 6. In Section 7, the conclusion of this paper is given.

2. Markov Decision Process and Reinforcement Learning

In this section, a brief introduction MDP and reinforcement learning (RL) is given. Being the fundamental model, MDP is often used to describe and solve the sequential decision-making problems involving uncertainties. In general, a model-free MDP is described as a tuple $\left(\mathbf{S},\mathbf{A},r,\gamma \right)$, where $\mathbf{S}$ is the state set in the decision process, $\mathbf{A}$ is the action set, r is the reward feedback by the environment after executing a certain action, and $\gamma$ is the discounted factor for measuring the current value of future return. The aim of MDP is to find the optimal policy that can maximize the cumulative discounted reward ${\displaystyle \sum _{k=0}^{\infty }}{\gamma }^k{r}_{t+k}$, and the policy function $\pi \left(\mathbf{a}\left|\mathbf{s}\right.\right)$ is expressed as follows:

$$\pi \left(\mathbf{a}\left|\mathbf{s}\right.\right)=p\left({\mathbf{A}}_t=\mathbf{a}\left|{\mathbf{S}}_t=\mathbf{s}\right.\right)$$

(1)

which is the probability of taking action $\mathbf{a}$ with current state $\mathbf{s}$.

Due to the randomness of the policy, the expectation operation is used to calculate the cumulative discounted reward function. When an action is given under a certain state, the cumulative discounted reward function becomes the state-action value function, i.e.,

$$Q\left(\mathbf{s},\mathbf{a}\right)=\mathbb{E}{}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k}\left|{\mathbf{S}}_t=\mathbf{s},{\mathbf{A}}_t=\mathbf{a}\right.\right]$$

(2)

which is also called Q function. This function is generally used in DRL for seeking the optimal policy.

In reinforcement learning (RL), there is an agent that can interact with the environment by actions, accumulate interactive data, and learn optimal action policy. In general, the commonly used reinforcement learning methods are classed into two kinds: one is based on the value function and the other is based on the policy gradient (PG). In value-function-based RLs, such as Q-learning, the learning criteria are looking for the action that can maximize the Q function by exploring the whole action space, i.e.,

$${\mathbf{a}}^{*}=arg\underset{\mathbf{a}}{max}Q\left(\mathbf{s},\mathbf{a}\right)$$

(3)

For the applications on continuous state and action space, value-function-based methods are not always available. Different from the circuitous way of Q-learning enumerating all action-value functions, policy-gradient-based methods find the optimal policy by directly calculating the update direction of policy parameters [21]. Therefore, the PG is more suitable for MDP problems over continuous action and state space. The goal of PG is to find the policy ${\pi }_{\vartheta }^{*}$ that can maximize the expectation of cumulative discounted reward, i.e.,

$${\pi }_{\vartheta }^{*}=arg\underset{{\pi }_{\vartheta }}{max}\mathbb{E}{}_{\tilde{\tau }\sim \pi }\left[G\left(\tilde{\tau }\right)\right]$$

(4)

where $\tilde{\tau }:{\mathbf{S}}_1,{\mathbf{A}}_1,{\mathbf{S}}_2,{\mathbf{A}}_2,{\mathbf{S}}_3,{\mathbf{A}}_3,{\mathbf{S}}_4,{\mathbf{A}}_4,\cdots \cdots$ represents the trajectory obtained by interacting with the environment using policy ${\pi }_{\vartheta }^{*}$, and $G\left(\tilde{\tau }\right)$ denotes the cumulative discounted reward on $\tilde{\tau }$. The loss function of the policy is defined as the expectation of the cumulative return, which is given by

$$J\left(\vartheta \right)={\mathbb{E}}_{\tilde{\tau }\sim \pi }\left[G\left(\tilde{\tau }\right)\right]={\int }_{\tilde{\tau }\sim {\pi }_{\vartheta }}{\pi }_{\vartheta }\left(\tilde{\tau }\right)G\left(\tilde{\tau }\right)d\tilde{\tau }$$

(5)

where ${\pi }_{\vartheta }\left(\tilde{\tau }\right)$ represents the occurrence probability of the trajectory $\tilde{\tau }$ with the policy ${\pi }_{\vartheta }$. The policy gradient can be obtained by deriving (5), as shown in the following:

$${\nabla }_{\vartheta }J\left(\vartheta \right)={\mathbb{E}}_{\tilde{\tau }\sim \pi }\left[{\nabla }_{\vartheta }log{\pi }_{\vartheta }\left(\tilde{\tau }\right)G\left(\tilde{\tau }\right)\right]$$

(6)

Compared with value-function-based methods, the PG-based methods has two advantages, one is higher efficiency in the applications of continuous action and state space, the other is that randomization of the policy can be realized.

3. Problem Statement

There are a monostatic MIMO radar and Q independent point targets in the MTT scenario. The monostatic MIMO radar performs both the transmitting task and receiving task. Assume that the location of MIMO radar is $\left(x_r,y_r\right)$, and the targets is initially located at $\left(x_0^q,y_0^q\right),q=1,2,3,\dots ,\mathtt{Q}$. At k-th tracking time step, the q-th target’s state is given by ${\mathbf{x}}_k^q={\left[x_k^q,{\dot{x}}_k^q,y_k^q,{\dot{y}}_k^q\right]}^T$, in which $\left[x_k^q,y_k^q\right]$ and $\left[{\dot{x}}_k^q,{\dot{y}}_k^q\right]$ denote the corresponding location and speed of q-th target, respectively. Besides that, the RCS components of targets can be also received by the radar at each time step, and that of the q-th target is denoted by ${\psi }_k^q={[h_{q,k}^R,h_{q,k}^I]}^T$, $h_{q,k}^R$ and $h_{q,k}^I$ are the RCS components of channel R and channel I, respectively. If we synthesize the target state component and the RCS component into a vector, an extended state vector can be obtained, i.e., ${\xi }_k^q={\left({\mathbf{x}}_k^{q^T},{\psi }_k^{q^T}\right)}^T$.

3.1. Derivation of BCRLB Matrix on Transmitting Parameters

Before modeling JBSPA, we first deduce the mathematical relationship between the tracking BCRLB and the beam selection parameters and power parameters in this section. According to [15], with the assumption that the beam number is the same as the target number, the expression of BIM is

$$J_m\left({\xi }_k^q\right)={\sum }_{{\xi }_k^q}^{-1}+{\mathbb{E}}_{{\xi }_k^q}\left[{\mathbf{D}}_{q,k}^{\mathrm{T}}{\sum }_{{\mathbf{y}}_k^q}^{-1}{\mathbf{D}}_{q,k}\right]$$

(7)

and its inverse is BCRLB matrix, i.e.,

$${\mathbf{B}}_{CRLB}\left({\mathbf{x}}_k^{}\right)=J_m^{-1}\left({\mathbf{x}}_k^{}\right)$$

(8)

In (7), the term ${\sum }_{{\xi }_k}$ represents the covariance matrix of the predictive state error, and its inverse matrix is the Fisher information matrix (FIM) of prior information. The term ${\mathbb{E}}_{{\xi }_k^q}\left[{\mathbf{D}}_{q,k}^{\mathrm{T}}{\sum }_{{\mathbf{y}}_k^q}^{-1}{\mathbf{D}}_{q,k}\right]$ is the FIM of the data. ${\mathbf{D}}_{q,k}^{}$ denotes the Jacobian matrix of the measurement ${\mathbf{y}}_k^q$ with respect to predictive target’s state ${\xi }_k^q$. The term ${\sum }_{{\mathbf{y}}_k^q}^{-1}$ represents the covariance matrix of the measurement noise [24,25].

It is known that FIM of the prior information only depends on the motion model of the target, while the FIM of the data is directly proportional to the transmitted power of the radar. Assume that the transmitted power is ${\mathbf{P}}_k={[P_{1,k},P_{2,k},\dots ,P_{\mathtt{Q},k}]}^T$ at time step k. When $P_{q,k}$ is zero, that is, there is no beam allocated to the q-th target for illumination at time step k, and the covariance matrix ${\sum }_{{\mathbf{y}}_k^q}^{-1}$ of the measurement error is a zero matrix. In order to describe the beam allocation more clearly, a group of binary variables ${\mathbf{z}}_k={[z_{1,k},z_{2,k},\dots ,z_{\mathtt{Q},k}]}^T$ is introduced here, where

$$z_{q,k}=\left\{\begin{array}{c}1,if{P}_{q,k}>0\hfill \\ 0,if{P}_{q,k}=0\hfill \end{array}\right.$$

(9)

where $z_{q,k}=1$ indicates that the radar illuminates target q at the k-th tracking time instant, and $z_{q,k}=0$ is the opposite. Then, substitute (9) into (7), and we obtain [26,27]

$$J_m\left({\xi }_k^q\right)={\sum }_{{\xi }_k^q}^{-1}+z_{q,k}{\mathbb{E}}_{{\xi }_k^q}\left[{\mathbf{D}}_{q,k}^{\mathrm{T}}{\sum }_{{\mathbf{y}}_k^q}^{-1}{\mathbf{D}}_{q,k}\right]$$

(10)

whose inverse matrix, BCRLB matrix, can be used as an evaluation function of transmitted power $P_{q,k}$ and beam selection parameter $z_{q,k}$. The expression is as follows:

$${\mathbf{B}}_{CRLB}\left(P_{q,k},z_{q,k}\right)\left|{}_{{\xi }_k^q}\right.={\left({\sum }_{{\xi }_k^q}^{-1}+z_{q,k}{P}_{q,k}{\mathbb{E}}_{{\xi }_k^q}\left[{\mathbf{D}}_{q,k}^{\mathrm{T}}{\widehat{\sum }}_{{\mathbf{y}}_k^q}^{-1}{\mathbf{D}}_{q,k}\right]\right)}^{-1}$$

(11)

in which the original covariance matrix of the measurement error ${\sum }_{{\mathbf{y}}_k^q}^{-1}$ is rewritten as a multiplication of $P_{q,k}$ and ${\displaystyle {\widehat{\sum }}_{{\mathbf{y}}_k^q}^{-1}}$.

3.2. Problem Formulation of JBSPA

In JBSPA, since the target state ${\xi }_k^q$ is unknown at the $k-1$-th time step, a predictive state ${\xi }_{k\left|k-1\right.}^q$ is taken to calculate the FIM of data. In addition, in order to ensure the requirement of real-time in MTT, we use one-sampling measurement to calculate the FIM of data instead of the expectation of FIM of data. The final BCRLB matrix is given by [14]

$${\mathbf{B}}_{CRLB}\left(P_{q,k},z_{q,k}\right)\left|{}_{{\xi }_{k\left|k-1\right.}^q}\right.={\left({\sum }_{{\xi }_k^q}^{-1}+z_{q,k}{P}_{q,k}{\widehat{\mathbf{D}}}_{q,k}^T{\widehat{\sum }}_{{\widehat{\mathbf{y}}}_k^q}^{-1}{\widehat{\mathbf{D}}}_{q,k}^{}\left|{}_{{\xi }_{k\left|k-1\right.}^q}\right.\right)}^{-1}$$

(12)

where ${\widehat{\mathbf{D}}}_{q,k}^{}$ denotes the Jacobian matrix of the measurement ${\widehat{\mathbf{y}}}_k^q$ with respect to predictive target’s state ${\xi }_{k\left|k-1\right.}^q$. ${{\displaystyle \widehat{\sum }}}_{{\widehat{\mathbf{y}}}_k^q}$ is the approximation of the covariance matrix of measurement at the predictive state ${\xi }_{k\left|k-1\right.}^q$. In theory, the diagonal elements of the derived BCRLB matrix represent the corresponding lower bounds of the estimation MSEs of the target states, and thus, the tracking BCRLB of the target is defined as the trace of the BCRLB matrix, i.e.,

$$\Phi \left(P_{q,k},z_{q,k}\right)=\mathrm{Tr}\left({\mathbf{B}}_{CRLB}\left(P_{q,k},z_{q,k}\right)\left|{}_{{\xi }_{k\left|k-1\right.}^q}\right.\right)$$

(13)

where $\mathrm{Tr}\left(·\right)$ represents trace operator. In order to enhance the MTT performance as much as possible, and ensure the tracking performance of each target, we take the worst-case tracking BCRLB among targets as an evaluation of the MTT performance to optimize the resource allocation result, which is expressed as

$$\mathrm{F}\left({\mathbf{P}}_k,{\mathbf{z}}_k\right)=\underset{q}{max}\sqrt{\mathsf{\Phi }\left(P_{q,k},z_{q,k}\right)}$$

(14)

The problem of JBSPA is then described as follows: restricted by the predetermined maximum beam number M and the power budget $P_{total}$, the power allocation and the beam selection are jointly optimized for the purpose of maximizing the cumulative discounted worst-case BCRLB of MTT. The cost function is written as

$$\begin{array}{cc}\underset{{\mathbf{P}}_k,{\mathbf{z}}_k}{\min }\hfill & {\displaystyle \sum _{u=k}^K}{\gamma }^{u-k}\mathrm{F}\left({\mathbf{P}}_k,z_k\right)\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & {\displaystyle \sum _{q=1}^{\mathtt{Q}}}{z}_{q,u}{P}_{q,u}=P_{total}\hfill \\ \hfill & {\delta }_k={\displaystyle \sum _{q=1}^{\mathtt{Q}}}{z}_{q,u}\le M,z_{q,u}\in \left\{0,1\right\},\hfill \\ \hfill & \left\{\begin{array}{c}{P}_{q,u}=0z_{q,u}=0\hfill \\ {\overline{P}}_{qmin}\le P_{q,u}\le {\overline{P}}_{qmax}{z}_{q,u}=1\forall u=k,\dots ,K,\forall q=1,\dots ,\mathtt{Q}\hfill \end{array}\right.\hfill \end{array}$$

(15)

where constraints 1∼3 are the total power constraint, the beam number constraint and the single beam power constraint, respectively. ${\overline{P}}_{qmin}$ and ${\overline{P}}_{qmax}$ are the minimum and the maximum of the single beam transmitted power, respectively. Due to the existence of binary beam selection parameters, the cost function (15) is a complex mixed-integer nonlinear optimization problem. As the discount factor is $\gamma =0$, the cyclic minimizing method [28] and projection gradient algorithm can be used to solve the problem [14]. However, since this method needs to traverse all possible beam allocation solutions to find the optimal solution, the solving process is time-consuming. With this method, if there are a large number of targets to track, the real-time performance of MTT cannot be guaranteed. On the other hand, when the discount factor $\gamma$ is non-zero, to solve the problem requires accurate future multi-step predictions of target state and BCRLB which is, however, hardly realized in real applications, especially in the scenarios involving maneuvering targets because the potential model-mismatch problem would amplify the prediction error and reduce the reliability of BCRLB seriously along with the tracking time increasing.

To address above issues, we discuss the JBSPA from MDP perspective, and solve it with a data-driven method: DRL. Compared with the optimization-based method, this method is able to avoid the multi-step prediction problem and complex solving process, and ensure the real-time performance of resource allocation. Besides that, in order to make the method suitable for maneuvering target scenarios, RNN with LSTM-GMM unit is used to learn the state transition regularity of maneuvering targets, and then realize the information prediction of the maneuvering target.

4. Joint Beam Selection and Power Allocation Strategy Based on Deep Reinforcement Learning

In this section, the original resource problem is first modeled as a model-free MDP. It is a complicated continuous controlling problem over action and state spaces. Therefore, the value-function-based DRLs, such as the DQN, are not applicable. For this problem, we choose to directly derive the Q function with respect to the policy parameters to calculate the policy gradient, and seek the optimal resource allocation policy along the descending direction of the policy gradient. In this paper, the proposed DRL-based method is named DRL–JBSPA (DRL for Joint Beam Selection and Power Allocation).

4.1. Overall

The workflow of the DRL–JBSPA system is shown in Figure 1. The implementation at each tracking interval, the current measurement of targets (i.e., radial range, angle, Doppler and RCS) will be imported into the DRL system to perform the tasks of the information prediction and the resource allocation. The prediction network has also been incorporated into the DRL system to provide the DRL agent state at each decision step. The key idea behind this design is to realize the best beam and power control by learning from the massive interaction data between the DRL agent and the environment. It is noticed that this environment here is not real, it is established artificially and consists of the agent state, action and reward. In this application, according to the objective and the resource limitations of JBSPA, the agent state, the action and the reward are designed in the following.

STATE: The state of the DRL agent is defined as

$${\mathbf{s}}_k={\left[{\xi }_{k\left|k-1\right.}^1,\dots ,{\xi }_{k\left|k-1\right.}^q,\dots ,{\xi }_{k\left|k-1\right.}^{\mathtt{Q}},diag\left({\sum }_{{\xi }_k^1}),\dots ,diag\left({\sum }_{{\xi }_k^q}\right),\dots ,diag({\sum }_{{\xi }_k^{\mathtt{Q}}}\right)\right]}^T$$

(16)

where $diag(·)$ represents the operation of taking the main diagonal elements. ${\sum }_{{\xi }_k^q}$ denotes the covariance matrix of the one-step predictive state ${\xi }_{k\left|k-1\right.}^q$. According to Section 3.1, the covariance matrix represents the prediction error of target state, and the agent can determine how to allocate resource by comparing the prediction error of all targets. Moreover, the predictive states of all of the targets are also included in the agent state vector, whose aim is to utilize the target state information, such as the targets’ ranges, the speeds and the RCS, to assist the agent in judgment. As $\gamma =0$, the DRL agent tends to allocate more power and beam resources to the targets with larger prediction errors to ensure their tracking performances [14].

ACTION: Instead of using a binary variable ${\mathbf{z}}_k$ to describe the beam allocation, two action parameters ${\delta }_k$ and ${\mathbf{X}}_k$ are taken here to represent the generated beam number and determine which target to be illuminated in the next tracking time instant, respectively. ${\delta }_k$ is a scalar and ${\mathbf{X}}_k$ is

$${\mathbf{X}}_k={\left[X_1,X_2,\dots ,X_{\mathtt{Q}}\right]}^T,X_q\in \left[0,1\right],q=1,2,\dots ,\mathtt{Q}$$

(17)

Then, synthesizing the power allocation action and the two beam allocation actions into an action vector, we have

$${\mathbf{a}}_k={\left[{{\delta }_k,{{\mathbf{X}}_k}^T,{\mathbf{P}}_k}^T\right]}^T$$

(18)

REWARD: An intuitive reward function will benefit the learning efficiency of DRL. According to the objective function in (15), we set the reward function to the opposite of the worst-case BCRLB at the k-th time step, i.e.,

$$r_k({\mathbf{s}}_k,{\mathbf{a}}_k)=-l\mathrm{F}\left({\mathbf{P}}_k,{\mathbf{z}}_k\right)$$

(19)

where l is a constant given artificially, and used for controlling the reward scale.

4.2. LSTM-GMM Based Prediction Network

The structure of prediction network is illustrated in Figure 2, where the LSTM is used with GMM output layer, say, LSTM-GMM. At the k-th time step, this network outputs the prior information representation ${\mathbf{V}}_k^q$ for target q with the measurement ${\mathbf{y}}_{k-1}^q$ as the input, which can be expressed in a mathematical form, i.e.,

$${\mathbf{V}}_k^q=f_{{\theta }^R}\left({\mathbf{y}}_{k-1}^q,h_{k-1}^q;{\theta }^R\right)$$

(20)

where $f_{{\theta }^R}(·)$ represents an abstracted function of the RNN-GMM and ${\theta }^R$ is its parameter vector. Based on maximum likelihood, the loss function is

$$L\left({\theta }^R\right)=-ln\left(p\left({\xi }_k^q\right)\right)$$

(21)

in which $p\left({\xi }_k^q\right)$ is the probability density function (PDF) of the prior distribution of the extended target state. Since the state vector and the target RCS component are independent of each other, the original PDF $p\left({\xi }_k^q\right)$ can be obtained by multiplying two independent PDFs, i.e.,

$$p\left({\xi }_k^q\right)=p\left({\mathbf{x}}_k^q\right)p\left({\psi }_k^q\right)$$

(22)

In this section, in order to describe the state transition of the maneuvering target and the randomness and uncertainty of the target RCS, we assume that the distributions of the target state and the target RCS are two different Gaussian mixture models that are composed of $K_t$ and $K_r$ Gaussian distributions, respectively. The PDFs are expressed as

$$p\left({\mathbf{x}}_k^q\right)=\sum _{k_t=0}^{K_t-1}{\mathsf{\Pi }}_{q,k}^{k_t}\frac{exp\left(-\frac{1}{2}{\left({\mathbf{x}}_k^q-{\left\{{\mu }_{{\mathbf{x}}_k}^q\right\}}^{k_r}\right)}^{\mathrm{T}}{\sum }_{{\mathbf{x}}_k^q}^{-1}\left({\mathbf{x}}_k^q-{\left\{{\mu }_{{\mathbf{x}}_k}^q\right\}}^{k_t}\right)\right)}{{\left(2\pi \right)}^2{\left|{\left\{{\sum }_{{\mathbf{x}}_k^q}^{}\right\}}^{k_r}\right|}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}},\sum _{k_t=0}^{K_t-1}{\mathsf{\Pi }}_{q,k}^{k_t}=1$$

(23)

$$p\left({\psi }_k^q\right)=\sum _{k_r=0}^{K_r-1}{\mathsf{\Pi }}_{q,k}^{k_r}\frac{exp\left(-\frac{1}{2}{\left({\psi }_k^q-{\left\{{\mu }_{{\psi }_k}^q\right\}}^{k_r}\right)}^{\mathrm{T}}{\sum }_{{\psi }_k^q}^{-1}\left({\psi }_k^q-{\left\{{\mu }_{{\psi }_k}^q\right\}}^{k_r}\right)\right)}{{\left(2\pi \right)}^2{\left|{\left\{{\sum }_{{\psi }_k^q}^{}\right\}}^{k_r}\right|}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}},\sum _{k_r=0}^{K_r-1}{\tilde{\mathsf{\Pi }}}_{q,k}^{k_r}=1$$

(24)

where ${\left\{{\mu }_{{\mathbf{x}}_k}^q\right\}}^{k_t}$ and ${\left\{{\sum }_{{\mathbf{x}}_k^q}^{}\right\}}^{k_t}$ are the mean and the covariance matrix of the $k_t$-th Gaussian distribution of target state ${\mathbf{x}}_k^q$, respectively. ${\left\{{\mu }_{{\psi }_k}^q\right\}}^{k_r}$ and ${\left\{{\sum }_{{\psi }_k^q}^{}\right\}}^{k_r}$ denote the mean and the covariance matrix of the $k_r$-th Gaussian distribution of RCS ${\psi }_k^q$, respectively. ${\mathsf{\Pi }}_{q,k}^{k_t}$ and ${\tilde{\mathsf{\Pi }}}_{q,k}^{k_r}$ are the corresponding mixed coefficients. Since the target state has four dimensions, the mean of the covariance matrix of $K_t$-th mixed Gaussian distribution can be expressed as follows.

$$\begin{array}{c}{\left\{{\mu }_{{\mathbf{x}}_k}^q\right\}}^{k_t}={\left[{\mu }_{x_{q,k}}^{k_t},{\mu }_{{\dot{x}}_{q,k}}^{k_t},{\mu }_{y_{q,k}}^{k_t},{\mu }_{{\dot{y}}_{q,k}}^{k_t}\right]}^{\mathrm{T}}\hfill \\ \\ {\left\{{\sum }_{{\mathbf{x}}_k^q}\right\}}^{k_t}=\left[\begin{array}{cccc}{\left({\sigma }_{x_k^q}^{k_t}\right)}^2& -{\rho }_1^{k_t}{\sigma }_{x_k^q}^{k_t}{\sigma }_{{\dot{x}}_k^q}^{k_t}& 0& 0\\ -{\rho }_1^{k_t}{\sigma }_{x_k^q}^{k_t}{\sigma }_{{\dot{x}}_k^q}^{k_t}& {\left({\sigma }_{{\dot{x}}_k^q}^{k_t}\right)}^2& 0& 0\\ 0& 0& {\left({\sigma }_{y_k^q}^{k_t}\right)}^2& -{\rho }_2^{k_t}{\sigma }_{y_k^q}^{k_t}{\sigma }_{{\dot{y}}_k^q}^{k_t}\\ 0& 0& -{\rho }_2^{k_t}{\sigma }_{y_k^q}^{k_t}{\sigma }_{{\dot{y}}_k^q}^{k_t}& {\left({\sigma }_{{\dot{y}}_k^q}^{k_t}\right)}^2\end{array}\right]\hfill \end{array}$$

(25)

In (25), it is assumed that the target state in x-direction and y-direction are independent from each other. The terms $\left[{\mu }_{x_{q,k}}^{k_t},{\mu }_{{\dot{x}}_{q,k}}^{k_t}\right]$ and $\left[{\sigma }_{x_k^q}^{k_t},{\sigma }_{{\dot{x}}_k^q}^{k_t}\right]$ are the mean and the standard deviation of the $K_r$-th Gaussian distribution of the target state on x-direction, respectively, and $\left[{\mu }_{y_{q,k}}^{k_t},{\mu }_{{\dot{y}}_{q,k}}^{k_t}\right]$ and $\left[{\sigma }_{y_k^q}^{k_t},{\sigma }_{{\dot{y}}_k^q}^{k_t}\right]$ denote that on y-direction. ${\rho }_1^{k_t}$ and ${\rho }_2^{k_t}$ are the correlation coefficients of the location and the speed on x-direction and y-direction, respectively. Since there are two channels of RCS, the PDF $p\left({\psi }_k^q\right)$ is a two-dimensional Gaussian, whose mean and covariance matrix are

$$\begin{array}{c}{\left\{{\mu }_{{\psi }_k}^q\right\}}^{k_r}={\left[{\mu }_{h_{q,k}^R}^{k_r},{\mu }_{h_{q,k}^I}^{k_r}\right]}^{\mathrm{T}}\hfill \\ \\ {\left\{{\sum }_{{\psi }_k^q}^{}\right\}}^{k_r}=\left[\begin{array}{cc}{\left({\sigma }_{h_{q,k}^R}^{k_r}\right)}^2& 0\\ 0& {\left({\sigma }_{h_{q,k}^I}^{k_r}\right)}^2\end{array}\right]\hfill \end{array}$$

(26)

where $\left({\mu }_{h_{q,Tk}^R}^{k_r},{\mu }_{h_{q,Tk}^I}^{k_r}\right)$ and $\left({\sigma }_{h_{q,Tk}^R}^{k_r},{\sigma }_{h_{q,Tk}^I}^{k_r}\right)$ represent the mean and the standard deviation of the $K_r$ Gaussian distribution on channel R and channel I, respectively.

Based on the above derivations and analysis, the output vector of RNN at the k-th time step is given in (27), and its dimension is $11K_t+5K_r$.

$${\mathbf{V}}_k^q={\left[{\mu }_{{\xi }_k^q}^{\mathrm{T}},{\sigma }_{{\xi }_k^q}^{\mathrm{T}},{\mathsf{\Pi }}_{{\xi }_k^q}^{\mathrm{T}},{\rho }_{{\mathbf{x}}_k^q}^{\mathrm{T}}\right]}^{\mathrm{T}}$$

(27)

where

$$\left\{\begin{array}{c}{\mu }_{{\xi }_k^q}^{}={\left[{\mu }_{x_{q,k}}^1,\dots ,{\mu }_{x_{q,k}}^{K_t},{\mu }_{{\dot{x}}_{q,k}}^1,\dots ,{\mu }_{{\dot{x}}_{q,k}}^{K_t},{\mu }_{y_{q,k}}^1,\dots ,{\mu }_{y_{q,k}}^{K_t},{\mu }_{{\dot{y}}_{q,k}}^1,\dots ,{\mu }_{{\dot{y}}_{q,k}}^{K_t},{\mu }_{h_{q,k}^R}^1,\dots ,{\mu }_{h_{q,k}^R}^{K_r},{\mu }_{h_{q,k}^I}^1,\dots ,{\mu }_{h_{q,k}^I}^{K_r}\right]}^{\mathrm{T}}\hfill \\ {\sigma }_{{\xi }_k^q}={\left[{\sigma }_{x_{q,k}}^1,\dots ,{\sigma }_{x_{q,k}}^{K_t},{\sigma }_{{\dot{x}}_{q,k}}^1,\dots ,{\sigma }_{{\dot{x}}_{q,k}}^{K_t},{\sigma }_{y_{q,k}}^1,\dots ,{\sigma }_{y_{q,k}}^{K_t},{\sigma }_{{\dot{y}}_{q,k}}^1,\dots ,{\sigma }_{{\dot{y}}_{q,k}}^{K_t},{\sigma }_{h_{q,k}^R}^1,\dots ,{\sigma }_{h_{q,k}^R}^{K_r},{\sigma }_{h_{q,k}^I}^1,\dots ,{\sigma }_{h_{q,k}^I}^{K_r}\right]}^{\mathrm{T}}\hfill \\ {\mathsf{\Pi }}_{{\xi }_k^q}={\left[{\mathsf{\Pi }}_{q,k}^1,\cdots ,{\mathsf{\Pi }}_{q,k}^{K_t},{\tilde{\mathsf{\Pi }}}_{q,k}^1,\cdots ,{\tilde{\mathsf{\Pi }}}_{q,k}^{K_r}\right]}^{\mathrm{T}}\hfill \\ {\rho }_{{\mathbf{x}}_k^q}={\left[{\rho }_1^1,\cdots ,{\rho }_1^{K_t},{\rho }_2^1,....,{\rho }_2^{K_t}\right]}^{\mathrm{T}}\hfill \end{array}\right.$$

(28)

In the implementation, at each tracking interval, we will randomly select one from the mixed Gaussian distribution outputs as the prior distribution of the extended target state according to the mixing coefficients. Assuming the $k_t$-th Gaussian distribution of target state and the $k_r$-th Gaussian distribution of RCS is selected, then the predictive extended target state and the corresponding FIM of prior information $J_P\left({\xi }_k^q\right)$ are given by

$${\xi }_{k\left|k-1\right.}^q={\left[{\mu }_{x_{q,k}}^{k_t},{\mu }_{{\dot{x}}_{q,k}}^{k_t},{\mu }_{y_{q,k}}^{k_t},{\mu }_{{\dot{y}}_{q,k}}^{k_t},{\mu }_{h_{q,k}^R}^{k_r},{\mu }_{h_{q,k}^I}^{k_r}\right]}^{\mathrm{T}}$$

(29)

$$J_P\left({\xi }_k^q\right)={\sum }_{{\xi }_k^q}^{-1}={\left[diagblk\left\{{\left\{{\sum }_{{\mathbf{x}}_k^q}^{}\right\}}^{k_t},{\left\{{\sum }_{{\psi }_k^q}^{}\right\}}^{k_r}\right\}\right]}^{-1}$$

(30)

where $diagblk(·)$ refers to the operation of the diagonal block matrix. Taking (29) and (30) into (13), we obtain the predicted tracking BCRLB of the q-th target at the k-th time step.

In the training stage, if the training set $\mathbf{D}={\left\{\left({\tilde{\mathbf{y}}}_1^m,{\xi }_1^m\right),\left({\tilde{\mathbf{y}}}_2^m,{\xi }_2^m\right),\cdots ,\left({\tilde{\mathbf{y}}}_{L_{\xi }}^m,{\xi }_{L_{\xi }}^m\right)\right\}}_{m=1}^{\mathrm{M}}$ consists of M tracks with length $L_{\xi }$, the loss of RNN is calculated by

$$L\left({\theta }^R\right)=\frac{1}{2}\sum _{m=1}^{\mathrm{M}}\sum _{k=1}^{L_{\xi }}{\left({\xi }_k^m-{\mu }_{{\xi }_k}^m\right)}^{\mathrm{T}}{\left({\sum }_{{\xi }_k}^m\right)}^{-1}\left({\xi }_k^m-{\mu }_{{\xi }_k}^m\right)$$

(31)

The time step of LSTM is set to $L_{\xi }$ and the Adam optimizer [29] is utilized with dropout. In the implementation, the probability of switching off the neuron output in the dropout function is set to 0.3.

As the prediction network structure shown in Figure 2, the structure of this network is established on a single-layer LSTM unit whose gate functions and nonlinear layer are all composed of 64 neurons. At the k-th time step, the observation vector is first normalized and then taken into the LSTM unit to execute “remember” and “forget” operations, and we obtain the final output ${\mathbf{O}}_k$ after the data flow through the hidden layer and the output layer, where ${\mathbf{O}}_k^{1\to 4K_t+2K_r}$ is the extended target state component, ${\mathbf{O}}_k^{4K_t+2K_r+1\to 8K_t+4K_r}$ is the standard deviation of each component of the extended target state, ${\mathbf{O}}_k^{8K_t+4K_r+1\to 9K_t+5K_r}$ are the mixed coefficients, and ${\mathbf{O}}_k^{9K_t+5K_r+1\to 11K_t+5K_r}$ are the correlation coefficients.

Since the standard deviation and the mixing coefficient of each target state component are strictly non-negative, the elements ${\mathbf{O}}_k^{4K_t+2K_r+1\to 8K_t+4K_r}$ and ${\mathbf{O}}_k^{8K_t+4K_r+1\to 9K_t+5K_r}$ should be operated as follows.

$${\widehat{\mathbf{O}}}_k^{4K_t+2K_r+1\to 8K_t+4K_r}=exp\left({\mathbf{O}}_k^{4K_t+2K_r+1\to 8K_t+4K_r}\right)$$

(32)

$${\widehat{\mathbf{O}}}_k^{8K_t+4K_r+1\to 9K_t+5K_r}=exp\left({\mathbf{O}}_k^{8K_t+4K_r+1\to 9K_t+5K_r}\right)$$

(33)

Additionally, considering that the output ${\mathbf{O}}_k$ is a standardized result, the final estimated target state is calculated using the mean and the standard deviation of the samples, i.e.,

$$\left\{\begin{array}{c}{\mu }_{{\xi }_k^q}={\mathbf{O}}_k^{1\to 4K_t+2K_r}\odot {\overline{\mathbf{s}}}_{\xi }+\overline{\xi }\hfill \\ {\sigma }_{{\xi }_k^q}^{}={\left({\widehat{\mathbf{O}}}_k^{4K_t+2K_r+1\to 8K_t+4K_r}\odot {\overline{\mathbf{s}}}_{\xi }\right)}_{}^2\hfill \\ {\mathsf{\Pi }}_{{\xi }_k^q}={\widehat{\mathbf{O}}}_k^{8K_t+4K_r+1\to 9K_t+5K_r}/\phantom{{\widehat{\mathbf{O}}}_k^{8K_t+4K_r+1\to 9K_t+5K_r}\sum _{}^{}{\widehat{\mathbf{O}}}_k^{8K_t+4K_r+1\to 9K_t+5K_r}}\sum _{}^{}{\widehat{\mathbf{O}}}_k^{8K_t+4K_r+1\to 9K_t+5K_r}\hfill \\ {\rho }_{{\mathbf{x}}_k^q}^{}={\mathbf{O}}_k^{9K_t+5K_r+1\to 11K_t+5K_r}\hfill \end{array}\right.$$

(34)

where ${\overline{\mathbf{s}}}_{\xi }$ and $\overline{\xi }$ are the mean and standard deviation of the samples, and the calculation is as follows.

$$\left\{\begin{array}{c}\overline{\xi }=\frac{1}{\mathrm{M}{L}_{\xi }}{\displaystyle \sum _{m=1}^{\mathrm{M}}}{\displaystyle \sum _{l=1}^{L_{\mathbf{x}}}}{\xi }_l^m\hfill \\ {\overline{\mathbf{s}}}_{\xi }=\sqrt{\frac{1}{\mathrm{M}{L}_{\xi }-1}{\displaystyle \sum _{m=1}^{\mathrm{M}}}{\displaystyle \sum _{l=1}^{L_{\mathbf{x}}}}{\left({\xi }_l^m-\overline{\xi }\right)}^2}\hfill \end{array}\right.$$

(35)

4.3. Actor-Critic-Based DRL for JBSPA

Considering that the MDP of JBSPA is a continuous controlling problem and the traditional Q-table method is not available, we adopt the PG-based method to solve it. By introducing the policy function ${\pi }_{\vartheta }\left(\mathbf{a}\left|\mathbf{s}\right.\right)$, the original problem (5) can be characterized as a classical MDP problem, which aims to find the optimal policy parameter ${\vartheta }^{*}$ that makes the Q-value function achieve the maximum, i.e.,

$${\vartheta }^{*}=arg\underset{\vartheta }{max}{Q}^{{\pi }_{\vartheta }}\left({\mathbf{s}}_k,{\mathbf{a}}_k\right)$$

(36)

where the Q function is calculated by

$$Q^{{\pi }_{\vartheta }}\left({\mathbf{s}}_k,{\mathbf{a}}_k\right)={\mathbb{E}}_{{\mathbf{s}}_{k+1}\sim {\rho }^{\pi },a_k\sim {\pi }_{\vartheta }\left({\mathbf{a}}_k\left|{\mathbf{s}}_k\right.\right)}\left[r\left({\mathbf{s}}_k,{\mathbf{a}}_k\right)+\gamma {\mathbb{E}}_{{\mathbf{a}}_{k+1}\sim {\pi }_{\vartheta }\left({\mathbf{a}}_{k+1}\left|{\mathbf{s}}_{k+1}\right.\right)}\left[Q^{{\pi }_{\vartheta }}\left({\mathbf{s}}_{k+1},{\mathbf{a}}_{k+1}\right)\right]\right]$$

(37)

in which the symbol ${\rho }^{\pi }$ refers to the state space with policy ${\pi }_{\vartheta }\left(\mathbf{a}\left|\mathbf{s}\right.\right)$. According to the definition of DRL state for our problem, we know that the DRL state is determined by the current target measurement information. However, the target measurement is always random duo to the existence of observation noise. Therefore, the DRL state transition is not deterministic, i.e.,

$$p\left({\mathbf{s}}_{k+1}\left|{\mathbf{s}}_k,{\mathbf{a}}_k\right.\right)\ne 1$$

(38)

According to (37), if the random policy is used at each decision time step, a large number of samples are required to calculate the Q value of all possible actions in the continuous action space or high-dimensional discrete action space. In order to reduce the computational complexity, the deterministic policy is adopted in this section, which is expressed as

$${\mathbf{a}}_k={\mu }_{\vartheta }\left({\mathbf{s}}_k\right)$$

(39)

where ${\mu }_{\vartheta }(·)$ is the deterministic policy function. Then, take (39) into (37), simplify, and have

$$Q^{{\mu }_{\vartheta }}\left({\mathbf{s}}_k,{\mathbf{a}}_k\right)={\mathbb{E}}_{{\mathbf{s}}_{k+1}\sim {\rho }^{\pi }}\left[r\left({\mathbf{s}}_k,{\mathbf{a}}_k\right)+\gamma Q^{{\pi }_{\vartheta }}\left({\mathbf{s}}_{k+1},{\mathbf{a}}_{k+1}\right)\right]$$

(40)

Next, derive the Q-value function with respect to $\vartheta$, and obtain the following deterministic policy gradient:

$${\nabla }_{\vartheta }{\mathcal{J}}^{\pi }\left(\vartheta \right)={\mathbb{E}}_{{\mathbf{s}}_{k+1}\sim {\rho }^{\pi }}\left[{\nabla }_{\vartheta }{\mu }_{\vartheta }\left({\mathbf{s}}_{k+1}\right){\nabla }_{\mathbf{a}}{Q}^{{\mu }_{\vartheta }}\left({\mathbf{s}}_{k+1},{\mathbf{a}}_{k+1}\right)\left|{}_{{\mathbf{a}}_{k+1}={\mu }_{\vartheta }\left({\mathbf{s}}_{k+1}\right)}\right.\right]$$

(41)

Subsequently, the equivalence between original problem and the established MDP problem is discussed in this section.

$$\begin{array}{c}\underset{\vartheta }{max}{Q}^{{\mu }_{\vartheta }}\left({\mathbf{s}}_k,{\mathbf{a}}_k\right)=\underset{\vartheta }{max}\left[{\mathbb{E}}_{{\mathbf{s}}_{k+1}\sim {\rho }^{\pi }}\left[r\left({\mathbf{s}}_k,{\mathbf{a}}_k\right)+\gamma Q^{{\pi }_{\vartheta }}\left({\mathbf{s}}_{k+1},{\mathbf{a}}_{k+1}\right)\right]\right]\hfill \\ =\underset{P_{q,k},q=1,\dots ,Q}{min}\left[\Phi \left(P_{q,k}\right)+\gamma \mathbb{E}\left[\sum _{m=k+1}^K{\gamma }^{m-k-1}\Phi \left(P_{q,m}\right)\right]\right]\hfill \\ =\underset{P_{q,k},q=1,\dots ,Q}{min}\mathbb{E}\left[\sum _{m=k+1}^K{\gamma }^{m-k}\Phi \left(P_{q,m}\right)\right]\hfill \end{array}$$

(42)

It can be seen from (42) that the MDP problem (36) is essentially the expectation form of original problem (15). In other words, the aim of the optimization problem is transformed to maximizing the expectation of the cumulative discounted MTT performance. The usage of the reinforcement learning makes it feasible to perform power allocation considering the long-term and short-term tracking performance by estimating the Q value to approximate the cumulative discounted MTT BCRLB.

Based on the derived deterministic policy gradient, this paper uses the deep deterministic policy gradient (DDPG) [22] algorithm to learn the optimal power allocation strategy based on the actor-critic framework. DDPG is a reinforcement learning algorithm that combines deep learning and the deterministic policy gradient. Its biggest advantage is that it can learn more effectively in the continuous action space. The structure of the DDPG mainly consists of two deep networks, i.e., the actor network and the critical network.

In the calculation of the Q value, the DDPG draws lessons from the idea of deep Q learning and adopts the target network with exactly the same structure as the critic-actor network to ensure the stability of network training. The DDPG prefers to update the target network parameters slightly by “soft” updating the target network, and achieve the improvement of learning stability. This operation is similar to the regular replication of parameters to the target network in the DQN. The soft update of the target network parameters is expressed as follows:

$$\begin{array}{c}{\theta }^{{\mu }^{\prime }}:=\tau {\theta }^{\mu }+\left(1-\tau \right){\theta }^{{\mu }^{\prime }}\hfill \\ {\theta }^{Q^{\prime }}:=\tau {\theta }^Q+\left(1-\tau \right){\theta }^{Q^{\prime }}\hfill \end{array}$$

(43)

where ${\theta }^{\mu }$ and ${\theta }^Q$ are the parameters of the actor network $\mu \left(\left.·\right|{\theta }^{\mu }\right)$ and the critic network $Q\left(\left.·\right|{\theta }^Q\right)$, respectively. ${\theta }^{{\mu }^{\prime }}$ and ${\theta }^{Q^{\prime }}$ are the parameters of the target networks ${\mu }^{\prime }\left(\left.·\right|{\theta }^{{\mu }^{\prime }}\right)$ and $Q^{\prime }\left(\left.·\right|{\theta }^{Q^{\prime }}\right)$, respectively. $\tau$ is the soft updating parameter which takes always a small value.

Exploration behavior is also introduced to make the DRL agent more experienced, with which the DRL agent is able to try unexplored areas to discover the improvement opportunity. At the k-th decision time step, the performed action is given by

$${\mathbf{a}}_k=\mu \left(\left.{\mathbf{s}}_k\right|{\theta }^{\mu }\right)+\mathcal{O}$$

(44)

where $\mathcal{O}$ is the random noise for agent exploration.

In order to further ensure the stability of learning, the experience relay technique is used to eliminate the strong temporal autocorrelation inherent to DRL, and can reduce the variance of the estimation of Q-values. At each training step, assuming that H experience samples are randomly selected from the experience replay buffer B to update the critic network and the actor network, the loss function of the critic network is written as

$$L^Q\left({\theta }^Q\right)=\frac{1}{\mathrm{H}}\sum _{\mathrm{j}=1}^{\mathrm{H}}{\left(Q\left({\mathbf{s}}_{\mathrm{j}},{\mathbf{a}}_{\mathrm{j}}\left|{\theta }^Q\right.\right)-Q_{\mathrm{j}}^{\mathrm{Tar}}\right)}^2$$

(45)

where $Q_{\mathrm{j}}^{\mathrm{Tar}}$ denotes the target Q value, and is used as “labels” for the estimation of the Q-values. $Q_{\mathrm{j}}^{\mathrm{Tar}}$ is calculated by

$$Q_{\mathrm{j}}^{\mathrm{Tar}}:=r_{\mathrm{j}}+\gamma Q^{\prime }\left(\left.{\mathbf{s}}_{\mathrm{j}},{\mu }^{\prime }\left(\left.{\mathbf{s}}_{\mathrm{j}}\right|{\theta }^{{\mu }^{\prime }}\right)\right|{\theta }^{Q^{\prime }}\right)$$

(46)

According to (41), the policy gradient with one sampling can be obtained by

$${\nabla }_{{\theta }^{\mu }}{\mathcal{J}}^{\pi }\left({\theta }^{\mu }\right)\approx \frac{1}{\mathrm{H}}\sum _{\mathrm{j}=1}^{\mathrm{H}}\left[{\nabla }_{{\theta }^{\mu }}\mu \left(\mathbf{s}\left|{\theta }^{\mu }\right.\right)\left|{}_{\mathbf{s}={\mathbf{s}}_{\mathrm{j}}}\right.{\nabla }_{\mathbf{a}}Q\left(\mathbf{s},\mathbf{a}\left|{\theta }^Q\right.\right)\left|{}_{\mathbf{s}={\mathbf{s}}_{\mathrm{j}},\mathbf{a}=\mu \left({\mathbf{s}}_{\mathrm{j}}\left|{\theta }^{\mu }\right.\right)}\right.\right]$$

(47)

The parameter of the actor network ${\theta }^{\mu }$ is synchronously updated based on the chain rule of gradient propagation. More details of the DRL–JBSPA are given in Algorithm 1.

Algorithm 1: DRL–JBSPA.
1:	Randomly initialize parameters ${\theta }^{\mu }$ and ${\theta }^Q$ of actor network $\mu \left(\left.·\right|{\theta }^{\mu }\right)$ and critic network $Q\left(\left.·\right|{\theta }^Q\right)$;
2:	Initialize target networks ${\theta }^{{\mu }^{\prime }}\leftarrow {\theta }^{\mu }$, ${\theta }^{Q^{\prime }}\leftarrow {\theta }^Q$;
3:	Initialize replay buffer B;
4:	for each episode do
4:	Initialize random process $\mathcal{O}$ for action exploration;
5:	Receive initial state ${\mathbf{s}}_1$ derived from the prediction network
	${\mathbf{s}}_1=f_{{\theta }^R}\left({\mathbf{y}}_0^q,h_0^q\right)$;
6:	for each step k of episode do
7:	Select action ${\mathbf{a}}_k=\mu \left(\left.{\mathbf{s}}_k\right|{\theta }^{\mu }\right)+\mathcal{O}$;
8:	Execute action ${\mathbf{a}}_k$ and observe state ${\mathbf{s}}_{k+1}$ and reward $r_k$
	${\mathbf{s}}_{k+1}=f_{{\theta }^R}\left({\mathbf{y}}_k^q\left({\mathbf{a}}_k\right),h_k^q\right)$
	$r_k=-\mathrm{F}\left({\mathbf{P}}_k,{\mathbf{z}}_k\right)$;
9:	Store transition sample $\left({\mathbf{s}}_k,{\mathbf{a}}_k,r_k,{\mathbf{s}}_{k+1}\right)$ into replay buffer B;
10:	Sample from B a minibatch of H transitions;
11:	Calculate the target Q value:
	$Q_{\mathrm{j}}^{\mathrm{Tar}}:=r_{\mathrm{j}}+\gamma Q^{\prime }\left(\left.{\mathbf{s}}_{\mathrm{j}},{\mu }^{\prime }\left(\left.{\mathbf{s}}_{\mathrm{j}}\right|{\theta }^{{\mu }^{\prime }}\right)\right|{\theta }^{Q^{\prime }}\right)$;
12:	Update critic network by minimizing the loss:
	$L^Q\left({\theta }^Q\right)=\frac{1}{\mathrm{H}}\sum _{\mathrm{j}=1}^{\mathrm{H}}{\left(Q\left({\mathbf{s}}_{\mathrm{j}},{\mathbf{a}}_{\mathrm{j}}\left|{\theta }^Q\right.\right)-Q_{\mathrm{j}}^{\mathrm{Tar}}\right)}^2$;
13:	Update the actor network using the sampled policy gradients:
	${\nabla }_{{\theta }^{\mu }}{\mathcal{J}}^{\pi }\left({\theta }^{\mu }\right)\approx \frac{1}{\mathrm{H}}\sum _{\mathrm{j}=1}^{\mathrm{H}}\left[{\nabla }_{{\theta }^{\mu }}\mu \left(\mathbf{s}\left|{\theta }^{\mu }\right.\right)\left|{}_{\mathbf{s}={\mathbf{s}}_{\mathrm{j}}}\right.{\nabla }_{\mathbf{a}}Q\left(\mathbf{s},\mathbf{a}\left|{\theta }^Q\right.\right)\left|{}_{\mathbf{s}={\mathbf{s}}_{\mathrm{j}},\mathbf{a}=\mu \left({\mathbf{s}}_{\mathrm{j}}\left|{\theta }^{\mu }\right.\right)}\right.\right]$;
14:	Update the target networks:
	${\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+\left(1-\tau \right){\theta }^{{\mu }^{\prime }}$
	${\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+\left(1-\tau \right){\theta }^{Q^{\prime }}$;
15:	end for
16:	end for

The used actor-critic network is shown in Figure 3, where the actor network contains two hidden layers and three output layers. The hidden layers are dense layers consisting of 256 neurons with ReLU activation, and the three output layers are all dense layers and share the same hidden layers. For different output actions, three output layers are named Actor 1, Actor 2 and Actor 3, respectively. Actor 1 consists of M neurons, and Actor 2 and Actor 3 are both composed of $\mathtt{Q}$ neurons.

At the k-th time step, the DRL state ${\mathbf{s}}_k$ is first input to the hidden layer in the actor network, and then, the output of the hidden layer will be taken into Actor 1. The activation function of Actor 1 is set as the softmax function $s_f\left(·\right)$, which is used to derive the probability of all possible beam numbers:

$${\mathbf{a}}^1=s_f\left({\mathbf{w}}_{A_1}^{}{\mathbf{h}}_k^A+{\mathbf{b}}_{A_1}^{}\right)$$

(48)

where ${\mathbf{h}}_k^A$ refers to the output of the hidden layer, ${\mathbf{w}}_{A_1}^{}$ and ${\mathbf{b}}_{A_1}^{}$ represent the weight and the bias of Actor 1, respectively. By selecting the superscript index corresponding to the maximum probability in ${\mathbf{a}}^1$, the number of beams generated by the radar at the k-th time step can be determined, i.e., ${\delta }_k=\arg \underset{}{\max }{\mathbf{a}}^1$.

Next, concatenate ${\mathbf{a}}^1$ with the output of the hidden layer, and then send it to Actor 2. Actor 2 is responsible for determining which target will be illuminated at the next tracking time step. For this purpose, the softmax activation function is used here to output the probability of each target illuminated. The output of Actor 2 is calculated by

$${\mathbf{a}}^2=s_f\left({\mathbf{w}}_{A_2}^{}\left[{\mathbf{h}}_k^A,{\mathbf{a}}^1\right]+{\mathbf{b}}_{A_2}^{}\right)$$

(49)

in which ${\mathbf{w}}_{A_2}^{}$ and ${\mathbf{b}}_{A_2}^{}$ are the weight and the bias of Actor 2. Then, the target to be illuminated can be determined by selecting the largest ${\delta }_k$ element from ${\mathbf{a}}^2$, which is ${\mathbf{X}}_k=\arg \underset{}{\max }\left({\mathbf{a}}^2,{\delta }_k\right)$.

In the same way, we concatenate the outputs of the hidden layer, actor 1 and actor 2, and import it into actor 3. Actor 3’s activation function is set to the tanh function. The calculation formula is as follows:

$${\mathbf{a}}^3=tanh\left({\mathbf{w}}_{A_3}^{}\left[{\mathbf{h}}_k^A,{\mathbf{a}}^1,{\mathbf{a}}^2\right]+{\mathbf{b}}_{A_3}^{}\right)$$

(50)

where ${\mathbf{w}}_{A_3}^{}$ and ${\mathbf{b}}_{A_3}^{}$ are the weight and the bias of Actor 3. Select the elements in ${\mathbf{a}}^3$ with index ${\mathbf{X}}_k$ to form the vector ${\mathbf{P}}_k$, and then normalize it to obtain the ratio of each target’s allocated power to the total transmitted power. So far, by selection and splicing the outputs of the three output layers, we finally obtain the complete action vector ${\mathbf{a}}_k={\left[{{\delta }_k,{{\mathbf{X}}_k}^T,{\mathbf{P}}_k}^T\right]}^T$.

The critic network takes the concatenated action vector ${\tilde{\mathbf{a}}}_k=\left[{\mathbf{a}}^1,{\mathbf{a}}^2,{\mathbf{a}}^3\right]$ and current agent state ${\mathbf{s}}_k$ as input and outputs the corresponding Q value. The critic network consists of two dense hidden layers and an output layer, where each of the hidden layers consists of 256 neurons with ReLU activation, and the output layer is a linear neuron. In our implementation, the Adam optimizer is used to train the actor-critic network, and the learning rate is set to 1e-4. The soft updating parameter $\tau$ is set to 1e-3. The deep learning library TensorFlow is taken to implement the networks on a personal computer with Intel Core i5-7500 CPU and 16GB RAM. NVIDIA GeForce GTX 1080TI GPU is used to accelerate the training process.

5. Simulation Results

In practice, the radar system has to face the challenge that the target number is larger than the beam number. Therefore, the simulation experiments are carried out under the condition that the maximum number of beams is less than the target number. Specifically, simulation 1 is conducted to investigate the proposed method by comparing the optimization-based method with the discounted factor $\gamma =0$, simulation 2 is taken to discuss the effect of the discounted factor $\gamma$ on the resource allocation results and the tracking performance.

The targets and the radar are depicted in the Cartesian coordinate system. Without loss of generality, the parameters of the radar system are set as follows: the radar system is assumed to be located at $\left(0\mathrm{km},0\mathrm{km}\right)$, the signal effective bandwidth and effective time duration of the qth beam are set as $B_{q,k}=2$ MHz and $T_{q,k}=1$ ms, respectively. The carrier frequency of each beam is set as 1 GHz, and thus, the carrier wavelength is ${\lambda }_q=0.3$ m. The time interval between successive frames is set as 2 s, and a sequence of 50 frames of data are utilized to support the simulation. The lower and upper bounds of the power of the qth beam are set to ${\overline{P}}_{qmin}=0.1P_{total}$ and ${\overline{P}}_{qmax}=P_{total}$, respectively. The maximum number of beams is 5 and the target number is 8. For better illustration of the effectiveness of the proposed method, the initial positions and initial velocities of the targets are set the same, i.e., $(1\mathrm{e}5\mathrm{m},1\mathrm{e}5\mathrm{m})$ and $\left(\pm 200\mathrm{m}/\mathrm{s},\pm 200\mathrm{m}/\mathrm{s}\right)$. The turning frequency of turn motion targets is set to 0.0035 Hz. The motion parameters of each target have been listed in

Table 1. Target motion parameters.

	Motion Mode	Turning Frequency	Initial Velocity
Target 1	Right-turn motion	0.0035	$\left(-200\mathrm{m}/\mathrm{s},-200\mathrm{m}/\mathrm{s}\right)$
Target 2	Left-turn motion	0.0035	$\left(-200\mathrm{m}/\mathrm{s},-200\mathrm{m}/\mathrm{s}\right)$
Target 3	Uniform linear motion	*	$\left(-200\mathrm{m}/\mathrm{s},-200\mathrm{m}/\mathrm{s}\right)$
Target 4	Right-turn motion	0.0035	$\left(-200\mathrm{m}/\mathrm{s},-200\mathrm{m}/\mathrm{s}\right)$
Target 5	Left-turn motion	0.0035	$\left(200\mathrm{m}/\mathrm{s},-200\mathrm{m}/\mathrm{s}\right)$
Target 6	Uniform linear motion	*	$\left(200\mathrm{m}/\mathrm{s},-200\mathrm{m}/\mathrm{s}\right)$
Target 7	Uniform linear motion	*	$\left(200\mathrm{m}/\mathrm{s},200\mathrm{m}/\mathrm{s}\right)$
Target 8	Left-turn motion	0.0035	$\left(-200\mathrm{m}/\mathrm{s},200\mathrm{m}/\mathrm{s}\right)$

, where the symbol “*” indicates that a motion mode does not include this motion parameter. The real moving trajectories of the three targets are depicted in Figure 4. As shown in Figure 4, Target 1 and Target 4 are doing right-turn motion, Target 2, Target 5 and Target 8 are doing left-turn motion; Target 3, Target 6 and Target 7 are doing uniform linear motion.

The prediction network is first trained for deriving prior information representation, where the training examples are the measurements of the targets’ states with the trajectories of left-turn motion, right-turn motion, and uniform linear motion, and the corresponding labels are true targets’ states. The initial position and the speed of the targets are randomly generated within [50 km, 150 km] and [−300 m/s, 300 m/s] on x-axis and y-axis, respectively. The turning frequencies are randomly generated within [0.001 Hz, 0.008 Hz]. In the process of generating training examples, the noise power added in measurements is determined by the allocated transmit power that is randomly generated within the interval $\left[{\overline{P}}_{qmin},{\overline{P}}_{qmax}\right]$. We trained RNN over 200,000 epochs, and AC-JBSPA over 500,000 epochs (i.e., 500,000 transition samples) in an off-line manner.

Simulation 1: In this section, a series of simulation experiments are carried out to validate the effectiveness of the aforementioned methods, in which we adopt root mean square error (RMSE) to measure tracking performance, and it is given by

$${\mathrm{RMSE}}_k=\sqrt{\frac{1}{N_{MC}}\sum _{j_m=1}^{N_{MC}}{∥{\mathbf{x}}_k^q-{\widehat{\mathbf{x}}}_{k\left|k\right.}^{q,j_m}∥}_2}$$

(51)

where $N_{MC}$ is the number of Monte Carlo trials, ${\parallel ·\parallel }_2$ denotes the 2-norm, and ${\widehat{\mathbf{x}}}_{k\left|k\right.}^{q,j_m}$ is the estimate of the qth target’s state at the $j_m$-th trial with Kalman filter [30].

We first compare the proposed method with the allocation methods introduced in [14] as the amplitude of target reflectivity is set to a constant 1. The optimization-based method is based on the optimization techniques, where the CV model is utilized for the prediction of target’s state and the calculation of the FIM of prior information. Since the optimization-based method only focuses on the performance of the current time instant, the discounted factor is set to $\gamma =0$ for a fair comparison. It is worth noting that the tracking RMSE of all the targets are set the same at the initial time, and in the figures about tracking performance, all of the RMSE curves start after the first resource allocation.

The comparison is shown in Figure 5, in which the curves labeled “${\mathrm{P}}_{\mathrm{Opt}}\left(\mathrm{RMSE}\right)$” and “${\mathrm{P}}_{\mathrm{DL}}\left(\mathrm{RMSE}\right)$” represent the tracking RMSEs averaged over 50 trails achieved with the optimization-based power allocation method, and proposed power allocation method, respectively. The curves labeled “${\mathrm{P}}_{\mathrm{Opt}}\left(\mathrm{BCRLB}\right)$” and “${\mathrm{P}}_{\mathrm{DL}}\left(\mathrm{BCRLB}\right)$” are the corresponding tracking BCRLBs. It can be seen that the tracking of the worst RMSEs are approaching the corresponding worst BCRLBs along with the number of measurements increasing. From the about 20th frame, the performance of the proposed method is about 10% higher than that of the optimization-based method with the CV model.

The estimation RMSEs and the corresponding power allocation results are presented in Figure 6 and Figure 7, respectively. In Figure 6a, the power allocation solution given by the proposed method makes the tracking RMSEs of different targets closer than that of the optimization-based method, which implies the resource utilization efficiency of the power allocation solution based on the learned policy (shown in Figure 7a) is higher than that of the optimization-based method with the CV model (shown in Figure 7b). In other words, there is room for the solution given by optimization-based method to continue to be optimized.

In Figure 7a,b, it can be seen that more power and beam resource tend to be allocated to targets that are moving away from radar, i.e., target 4, target 5, target 6, and target 7, for achieving a larger gain of tracking performance. Actually, the resource allocation is not only determined by targets’ radial distances but also by radius speeds, especially when the targets are close enough. As shown in Figure 7a, comparing the allocation results of these targets that are approaching the radar, e.g., Target 1, Target 2, Target 3, and Target 8, we discover that more power tends to be allocated to the beam pointing to the closer Target 1 because Target 1’s larger radial speeds may cause a larger BCRLB. When using the CV model for prediction, the beam selection and the power allocation are mainly determined by comparing the radius distances of different targets. For example, for the resource allocation solution of Target 1, Target 2, Target 3 shown in Figure 7b, more power and beam resources tend to be allocated to the farther targets, Target 1 and Target 2, from about frame 20. Therefore, we believe the proposed method can provide more accuracy prior information of targets, which benefits the utilization effectiveness of resource allocation.

To better illustrate the effectiveness of the RNN-based prediction network, an additional simulation experiment is conducted, where the optimization-based method is used to solve cost function (5) based on the prior information representation derived from the prediction network. In Figure 8, the curve labeled “${\mathrm{P}}_{\mathrm{Opt}}^{\mathrm{RNN}}\left(\mathrm{RMSE}\right)$” represents the worst case tracking RMSE of the RNN-optimization-based method. It can be seen that this curve almost coincides with that of the proposed data-driven method and is also higher than that of the optimization-based method. This shows that the cost function (5) can be optimally solved with the method in [14] once a one-step predictive state is accurate enough. According to the comparison shown in Figure 8, it is known that, as $\gamma =0$, the tracking performance improvements brought from the proposed method are mainly due to the usage of the RNN-based prediction network.

Furthermore, a prediction performance comparison of the RNN-based prediction network and the Kalman filter is given in Figure 9, where the power is fixed during tracking. We first compare the RMSE of the predictive state of Target 2 (doing left turns) with the RNN and Kalman filter. The results are shown in Figure 9a, in which the prediction performance of RNN is higher than that of Kalman filter with the CV model because the model mismatch problem is avoided to a certain extent. Additionally, Figure 9b shows a comparison of the two methods of Target 3 (doing uniform linear motion). Here, Target 3 is taken in order to avoid the influence of model-mismatch. It can be seen that, different from that of Target 2, the RMSE curves of the two methods almost coincide, which implies that the performance of the RNN-based prediction network is very close (almost the same) to that of the Kalman filter with the true model (CV model), and thus, we believe that the RNN-based prediction network is able to replace the model-based prediction for target tracking in the applications of radar system resource allocation.

Next, we discuss the two methods with given RCS models. The amplitude sequences of RCS are given in Figure 10. In Figure 11, the curves labeled “${\mathrm{P}}_{\mathrm{Opt}}\left(\mathrm{RMSE}\right)$” and “${\mathrm{P}}_{\mathrm{DL}}\left(\mathrm{RMSE}\right)$” are the worst tracking RMSEs averaged over 50 trails achieved with the optimization-based method and the proposed method, respectively. Similar to that in the first case, the tracking performance of the proposed data-driven-based method is also superior to that of the optimization-based method. In Figure 12 and Figure 13, we present the tracking RMSE and the corresponding resource allocation results of each target with the two methods. In Figure 12, the tracking RMSE curves of targets given by the proposed method is closer than that of the optimization-based method. This indicates that the resource utilization efficiency of the optimization-based method is lower than that of the proposed method. Comparing the resource allocation results of Target 1, Target 2 and Target 3 in Figure 13a, we observe that the revisit frequency of radar for Target 3 is higher than that of the other two. This phenomenon can be explained in that a high revisit frequency benefits the tracking performance improvement of the targets with a larger radial speed. On the other hand, more power and beam resource tend to be allocated to these targets with smaller RCS amplitude, i.e., Target 1, Target 2 and Target 3, which aim to improve SNR of echoes such that the worst-case tracking performance among multiple targets is maximized.

Simulation 2: In this section, we investigate the effect of the discounted factor $\gamma$ on the resource allocation results and the tracking performance. Theoretically, taking cumulative discounted tracking accuracy into the resource allocation problem is more reasonable, especially as maneuvering targets are involved. Without loss of generality, three conditions are discussed, i.e., $\gamma =0$, $\gamma =0.5$ and $\gamma =0.99$. The target parameters are set the same as that in simulation 1 and the amplitude of the target reflectivity is also set to 1.

The worst-case tracking RMSEs with different $\gamma$ are shown in Figure 14, where the result with $\gamma =0.99$ is lower than that with $\gamma =0$ and $\gamma =0.5$ at the beginning, and later surpasses them with frame number increasing. This phenomenon is consistent with that the discounted factor controls the importance of the future tracking accuracy on the current resource allocation solution. Particularly, if taking a small discounted factor, the learned policy tends to improve the performance of the nearer future tracking time step at a cost of a performance loss of the farther future tracking time. In Figure 15, the tracking RMSE curves with $\gamma =0.5$ are more compact than that with $\gamma =0.99$ before frame 30 in order to reduce the worst-case tracking RMSE of the nearer future tracking time, which is the opposite after frame 30. The corresponding resource allocation results are presented in Figure 16. In Figure 16a, more power and beam resource are allocated to the beam pointing to target 4, target 5, target 6, and target 7, which are moving away from radar such that a larger gain of future tracking performance can be achieved. Meanwhile, increasing the revisit frequency of Target 3 aims to preventing from future larger BCRLB due to its larger radial speeds, which is the same as the result of $\gamma =0$. By comparing the tracking performance curves in Figure 14, it is easy to see that the tracking RMSE curves of non-zero discounted factors are lower than that of $\gamma =0$ within frame [30, 50], which indicates that the usage of non-zero discounted factors makes the resource allocation policy improve the performance of farther tracking time. In this case, as shown in Figure 16b, in order to improve the worst-case tracking RMSE of farther tracking time as $\gamma =0.99$, Target 7 (it moves away from the radar with the largest radial speed) is illuminated all the time even at the beginning of tracking, and takes up most of the power at the end of tracking. This phenomenon is different from the results with $\gamma =0$ that cares more about nearer future tracking performance.

In all, the usage of a larger $\gamma$ contributes to a more stable and effective allocation policy in the case of considering farther future tracking performance. However, in this case, a performance loss of the early tracking stage would not be avoided. In other words, there is a trade-off between the current tracking performance and the future tracking performance about $\gamma$.

6. Discussion

This paper focuses on the problem of radar power allocation for multi-target tracking from a new perspective. A novel cognitive resource allocation framework is established based on deep learning. Experimental results show that the proposed method is suitable for applications that need to balance the long-term and short-term multi-target tracking performance under limited resources. In practical applications, such as tracking aircraft or missiles, targets often maneuver during tracking process, and the motion models are always unknown due to uncertainty of target motion [8,31]. In this case, the usual way is to use alternative models [32,33], such as CV and CA, for target information prediction including target position, RCS and BCRLB. However, there is a large prediction error in the prediction stage due to the existing model mismatch, which will reduce the resource allocation performance and target tracking performance. Compared with the model-optimization-based methods, the proposed data-driven method can avoid the usage of an alternative model in the prediction stage, and provide more reliable target information for resource allocation. The adoption of the combination of RNN and DRL is able to make the radar system more flexible, intelligent and forward-looking. On the other hand, since the deep-learning-based method does not need to have specific optimization methods for specific problems, we believe that this kind of data-driven method is able to solve most types of radar resource allocation problems in target tracking, such as resource allocation in scenarios of multiple maneuvering target tracking, the resource allocation considering short-term tracking effect or long-term tracking effect, and resource allocation for multiple radar system (MRS).

Our future work will mainly focus on following two aspects:

(1)

Establishing a richer maneuvering dataset. In the simulation experiments of this work, in addition to the uniform motion, the motion dataset of the prediction network also contains the data of some common maneuvering models, such as left-turn motion, right-turn motion, uniform acceleration motion, and so on. However, in real applications, the target maneuvering pattern could be more complex, and there may be the problem of target missing and the association of multiple targets. Therefore, the establishment of a more complete maneuvering target dataset is very important for a cognitive radar system.

(2)

Integrating various types of resource allocation networks and establishing a more intelligent resource allocation system. In theory, the cognitive resource allocation framework based on deep learning proposed in this paper can be used for resource allocation in many scenarios. However, resource allocation networks still need to be trained for different scenarios and different requirements, respectively, such as the scenarios with different target numbers and radar numbers. In practical applications, scenarios and requirements may change rapidly, and thus, the resource allocation strategy needs to be adjusted in real time. How to make the radar system intelligently select the appropriate resource allocation network with the changing of the scenarios and the requirements is of great significance. In our future work, the aim is to unify various types of resource allocation networks to establish an organic system and make the resource allocation process more intelligent.

7. Conclusions

In this paper, a data-driven JBSPA method for MTT is proposed and the simulation results have demonstrated its superiority. There are two keys to enhancing the MTT performance and the resource utilization efficiency, one is using the LSTM-GMM for target information prediction, which provides a reliable prior information for resource allocation, the other one is forming an MDP of resource allocation and solving with the DRL, which realizes complicate resource allocation considering long-term tracking performance. In summary, this method integrates the decision-making ability of DRL and the memory and prediction ability of RNN to make the resource allocation more flexible and further improve the cognitive ability of the radar system.