Constellation-Optimized IM-OFDM: Joint Subcarrier Activation and Mapping via Deep Learning for Low-PAPR ISAC

Abstract

Orthogonal frequency division multiplexing (OFDM) has been regarded as an attractive waveform for integrated sensing and communication (ISAC). However, suffering from its high peak-to-average power ratio (PAPR), sensitivity to phase noise (PN), and spectral efficiency saturation, the performance of OFDM in ISAC is limited. Against this background, this paper proposes a constellation-optimized index-modulated OFDM (CO-IM-OFDM) framework that leverages neural networks to design a constellation suitable for subcarrier activation patterns. A correlation model between index modulation and constellation is established, enabling adaptive constellation mapping in IM-OFDM. Then, Adam optimizer is employed to train the constellation tailored for ISAC, enhancing spectral efficiency under PN and PAPR constraints. Furthermore, a weighting factor is defined to characterize the joint communication–sensing performance, thus optimizing the overall system performance. Simulation results demonstrate that the proposed method can achieve improvements in bit error rate (BER) by over 4 dB and in Cramér–Rao bound (CRB) by 2% to 8% compared to traditional IM-OFDM constellation mapping. It overcomes fixed constellation constraints of conventional IM-OFDM systems, offering theoretical innovation waveform design for low-power communication–sensing systems in highly dynamic environments.

1. Introduction

Integrated sensing and communication (ISAC) technology has witnessed rapid development and found increasingly extensive applications in the sixth generation (6G) networks, intelligent transportation systems, and internet of things (IoT) [1,2]. Given the precious and scarce broadband resources, researchers are actively investigating efficient spectrum utilization strategies. Consequently, dual-functional waveforms that integrate wireless communication and sensing capabilities have emerged as a critical frontier in 6G [3,4]. How to design efficient and channel-adaptive dual-functional waveforms has become a pivotal challenge demanding urgent resolution, particularly under dynamic channel conditions.

Thanks to its high spectral efficiency and multipath interference resistance, conventional OFDM dominates fifth generation (5G) implementations. However, the high peak-to-average power ratio (PAPR) and acute sensitivity to phase noise (PN) of OFDM typically limit its practical implementation [5,6,7,8]. In contrast, index-modulated orthogonal frequency division multiplexing (IM-OFDM), which represents a dual-functional waveform technology, has shown emerging potential for ISAC applications [9,10]. In particular, benefiting from its dynamic activation of the subcarrier, IM-OFDM enables low-power transmission [11]. IM-OFDM encodes information not only through conventional quadrature amplitude modulation (QAM) symbols, but also through the indices of activated subcarriers in the frequency domain [12,13]. While index modulation grants IM-OFDM advantages, including superior energy efficiency, reduced PAPR, and robustness against inter-carrier interference (ICI), the selective subcarrier activation procedure introduces inherent spectral efficiency compromises compared to traditional OFDM [14].

Adaptive constellation design for IM-OFDM can effectively enhance spectral efficiency. For example, the authors in [15] proposed dual-mode OFDM-IM (DM-OFDM-IM), which partitions subcarriers within subblocks into two groups and employs different constellation mappings for each group to improve transmission rates. However, conventional constellations exemplified by phase shift keying (PSK) and QAM induce significant power penalty due to the increased PAPR. To tackle this problem, the authors in [16,17] proposed distinct constellation pairs to reduce pairwise error probability (PEP) and bit error rate (BER) in DM-OFDM-IM systems. The authors in [18] introduced transmit diversity in IM-OFDM through silent subcarriers and multi-signal constellations, which further improved power efficiency. Nevertheless, existing IM-OFDM constellation designs predominantly employ conventional constellation combinations, which achieve a mild throughput improvement while fundamentally suffering critical limitations of PAPR and PN [19,20].

Recently, artificial intelligence (AI) has shown significant potential in adaptive constellation generation tailored to IM-OFDM under stringent constraints [21,22,23,24]. In contrast to conventional constellation designs, which are based on the assumptions of ideal Gaussian channels while ignoring the effect of hardware impairments such as nonlinearities of the power amplifier (PA) [25], the AI-enabled constellation design shows the ability to adapt to such impairments. More specifically, with the help of neural networks, the transceiver of wireless communication systems is represented as a specialized neural network capable of learning symbol mapping processes and channel characteristics, thus reconstructing signals with minimal distortion at the receiver [26,27,28]. Ref. [20] proposes an OFDM modulation recognition (MR) architecture based on a scattering transform MR-network, considering constraints of radio frequency nonlinear distortion. Ref. [29] employs a blind channel equalization strategy to mitigate the effects of fading channels. Ref. [30] classifies OFDM symbol modulation based on convolutional neural networks. Implementing constellation mapping through the neural network structure enables the discovery of optimal symbol constellations that balance spectral efficiency, power efficiency, and PN robustness [31,32].

Against this background, this paper proposes a neural network-empowered constellation-optimized IM-OFDM (CO-IM-OFDM) architecture, which can derive optimal constellations for index-modulated subcarriers under dual constraints of low PAPR and PN resilience. The framework jointly optimizes communication and sensing performance through weighted metric training. The main contributions are summarized as follows:

• A neural network-based CO-IM-OFDM framework is proposed, which jointly optimizes subcarrier indices and constellation diagrams. By incorporating PAPR constraints, the trained constellations are suitable for IM-OFDM systems.
• A neural network tailored for IM-OFDM constellations in ISAC systems is proposed. Using Adam optimizer, the network generates constellations considering PN robustness and PA nonlinearity constraints, which jointly optimize spectral efficiency and ranging accuracy.
• We analyze constellation designs and performance metrics under varying trade-off weights via numerical simulations. With bit error rate (BER) and Cramér–Rao bound (CRB) as performance metrics, a weighting factor is defined to characterize joint performance, enabling maximization of the overall system efficiency.

2. Fundamentals

2.1. IM-OFDM Module

The block diagram of the considered IM-OFDM system is presented in Figure 1. Let the total available subcarriers be M with subcarrier spacing $\Delta f$ and symbol duration $T_s$. For implementation convenience, these subcarriers are divided into G groups, each containing $N_g=M/G$ subcarriers. Since each subblock undergoes identical procedures, it suffices to consider the $\xi$-th subblock ($\xi \in \{1,\dots , G\}$). Within each group, there are $N_g$ subcarriers with indices $(i_1,\dots , i_n,\dots , i_{N_g})$, where $i_n$ is the index of the n-th subcarrier in each subblock.

In the IM-OFDM system, k subcarriers in each subblock are activated to transmit modulated symbols, where the indices of the activated subcarriers are stored in the set ${\mathit{I}}_g$, and the complex modulated symbol set is $\mathit{S}$. An example look-up table with a subblock of $N_g=4, k=2$ with symbols $C_1, C_2$ is shown in Figure 1. Assume the input bitstream of N bits is partitioned into $N_1$ and $N_2$ bits, which are given by

$$N_1=⌊{log}_2\left({\displaystyle \genfrac{}{}{0pt}{}{N_g}{k}}\right)⌋, N_2=k{log}_2\left(Q\right),$$

where $\left({\displaystyle \genfrac{}{}{0pt}{}{n}{k}}\right)$ denotes the binomial coefficient representing the number of k-element combinations from n elements, and Q is the modulation order defined by the constellation diagram. $N_1$ bits determine indices of activated subcarriers, and the remaining $N_2$ bits form the information sequence ${\mathit{b}}_B\left(n\right)={[b_B\left(1\right),\dots , b_B\left(N_2\right)]}^T$, which is mapped via constellation modulation to generate the symbol sequence $\mathit{c}\left(n\right)={[c\left(1\right),\dots , c\left(N_c\right)]}^T$, where $c\left(n\right)\in \mathit{S}$.

Subsequently, during the j-th OFDM symbol ($j=1,\dots , N_s$), where $N_s$ is the length of the OFDM symbol, its communication-oriented signal is expressed as

$${\mathit{x}}_c\left(j\right)={[x_{c,1}\left(j\right),\dots , x_{c,m}\left(j\right),\dots , x_{c,M}\left(j\right)]}^T,$$

(1)

where $x_{c,m}\left(j\right)\in \mathit{S}$ if $m\in {\mathit{I}}_g$; otherwise, $x_{c,m}\left(j\right)=0$. The total number of information bits per symbol is given by $N_1+N_2$ bits.

Different from conventional approaches to realizing sensing functionality, which insert communication and sensing signals into separate subcarriers or superimpose both onto index-modulated subcarriers [13], the sensing signal ${\mathit{x}}_s\left(j\right)={[x_{s,1}\left(j\right),\dots , x_{s,M}\left(j\right)]}^T$ in our considered scheme employs pseudo-random sequences with superior auto-correlation properties. The sequences satisfy $\mathbb{E}\left[{\mathit{x}}_c^H\left(j\right){\mathit{x}}_s\left(j\right)\right]=0$, indicating statistical orthogonality between communication and sensing signals. Through the power allocation factor $\rho$, we linearly combine them to maintain consistent transmission power, yielding the transmitted signal ${\mathit{x}}_F\left(j\right)={\mathit{x}}_s\left(j\right)+{\mathit{x}}_c\left(j\right)$, where ${\mathit{x}}_F\left(j\right)={[x_{F,1}\left(j\right),\dots , x_{F,M}\left(j\right)]}^T$, which can guarantee both communication and sensing functionalities.

By performing inverse fast Fourier transform (IFFT) on the frequency-domain samples ${\mathit{x}}_F$, the time-domain OFDM symbol is given by

$$\mathit{x}\left(j\right)=\mathrm{IFFT}\left\{{\mathit{x}}_F\left(j\right)\right\}={\left[x_1\left(j\right),\cdots , x_m\left(j\right),\cdots , x_M\left(j\right)\right]}^T, (m=1,\dots ,M, j=1,\dots ,N_s)$$

(2)

where $x_m\left(j\right)$ is the element of the OFDM symbol matrix ${\mathit{X}}_{M\times N_s}\in {\mathbb{C}}^{M\times N_s}$. After IFFT operation, a cyclic prefix (CP) of length C is appended to the beginning of the OFDM symbol, given by

$${\mathit{x}}_{\mathrm{cp}}\left(j\right)={\left[x_{M-C+1}\left(j\right),\cdots , x_M\left(j\right),x_1\left(j\right),x_2\left(j\right),\cdots x_M\left(j\right)\right]}^T,$$

(3)

where ${\mathit{x}}_{\mathrm{cp}}\in {\mathbb{C}}^{(M+C)\times N_s}$. Then, a parallel-to-serial (P/S) conversion is performed to sequentially concatenate IM-OFDM symbols, forming the transmitted signal ${\mathit{x}}_{\mathrm{t}}\in {\mathbb{C}}^{N_s(M+C)\times 1}$. Subsequently, the baseband discrete signal undergoes digital-to-analog conversion (DAC) and is upconverted to radio frequency (RF) band before transmission. The passband analog signal can be expressed as

$${\mathit{x}}_{\mathrm{rf}}=\Re \left\{{\mathit{x}}_{\mathrm{t}}{e}^{j(2\pi f_{c}t+{\varphi }_c)}\right\},$$

(4)

where $f_c$ and ${\varphi }_c$ are the carrier frequency and initial phase of transmitted signal, respectively. Then, the signal is sent via the Gaussian fading channel.

2.2. Operation of IM-OFDM Receiver

At the IM-OFDM receiver, the received signal is processed to achieve both target sensing and communication decoding. We consider one frame of OFDM data; the received signal in the frequency domain after passing through the channel can be expressed as

$$\mathit{y}\left(j\right)=\mathit{x}\left(j\right){\mathit{h}}_{\mathit{F}}\left(j\right)+{\mathit{w}}_{\mathit{F}}\left(j\right),j=1,\dots , N_s,$$

(5)

where $\mathit{y}={[y_1,\dots , y_M]}^T$, ${\mathit{h}}_{\mathit{F}}={[h_{F,1},\dots , h_{F,M}]}^T$, and ${\mathit{w}}_{\mathit{F}}={[w_{F,1},\dots , w_{F,M}]}^T$ are the received signal, channel fading coefficient, and noise sample in the frequency domain, respectively. ${\mathit{w}}_{\mathit{F}}$ is additive white Gaussian noise (AWGN) with zero mean and variance ${\sigma }^2$. Regarding target sensing, we can handle ${\mathit{x}}_c$ and ${\mathit{x}}_s$ separately thanks to the statistical orthogonality between them. Considering time delay $\tau$ and Doppler shift $f_D$, the channel function can be expressed as ${\mathit{h}}_{\mathit{F}}=H_p{e}^{-\mathrm{j}2\pi m\Delta f\tau }{e}^{\mathrm{j}2\pi f_{D}j{T}_s}$, where $H_p$ is the attenuation due to the pass loss, scattering and radar cross-section. By correlating $\mathit{y}$ with ${\mathit{x}}_s$ at the estimated value, the estimate of $\tau$ is identified as the peak of range power spectrum.

For bit decoding, we analyze received signal by subtracting the sensing component. The indices of activated subcarriers and their corresponding information symbols can be detected. In this process, we employ detection algorithms for IM-OFDM, specifically using the Log-Likelihood Ratio (LLR) detector. The posterior probability ratio of frequency-domain symbols being non-zero versus zero provides information about the activation status of index m, which is given by

$$\lambda \left(m\right)=ln{\displaystyle \frac{{\sum }_{i=1}^{M}P(x_m=c\left(i\right)|y_m)}{P(x_m=0|y_m)}},$$

(6)

where $c\left(i\right)\in \mathcal{S}$. A higher $\lambda \left(m\right)$ value indicates a greater likelihood of index m being activated by the transmitter’s index selector. Based on Bayes’ theorem, we get

$$\lambda \left(m\right)=ln\left({\displaystyle \frac{k}{n-k}}\right)+{\displaystyle \frac{|y_m{|}^2}{N_0}}+ln\left(\sum _{i=1}^{M}exp\left(-{\displaystyle \frac{{|y_m-h_{F,m}c\left(i\right)|}^2}{N_0}}\right)\right),$$

(7)

where $N_0$ is the noise variance in the frequency domain. With higher-order modulations, we can define

$$ln\left(\sum _{i=1}^M{e}^{a_i}\right)=f_{\max }(\cdots f_{\max }(f_{\max }(a_1,a_2),a_3),\dots ,a_M),$$

(8)

where $f_{\max }(a,b)=ln(e^a+e^b)=max(a,b)+ln(1+e^{-|a-b|})$.

After computing M LLR values, the receiver selects k indices with maximum LLR values per subblock. The detected indices are then passed to an index demapper to estimate $N_1$ bits, followed by standard constellation demodulation.

3. Proposed Method

3.1. Framework of Constellation Optimization via Neural Network

Reliable transmission in IM-OFDM systems requires the consideration of PN and PA nonlinearity constraints. Therefore, this subsection integrates the IM-OFDM transmission system with mapping optimization and loss function, presenting a neural network-based constellation optimization system architecture for IM-OFDM.

In traditional communication systems, constellation design typically follows predefined rules, where constellations such as QAM and PSK are adopted. These constellation designs usually rely on theoretical assumptions of ideal Gaussian channels, thus making them ill-suited to dynamic channel environments and nonlinear PA constraints. In contrast, deep learning can autonomously learn the correlation between channel characteristics and transmission objectives through input–output architectures. More importantly, modeling the constellation mapping process as a trainable neural network layer enables end-to-end optimal mapping training [33]. To be noticed, such a neural network-empowered communication system can learn robust message representations associated with channel impairments during symbol mapping, thereby facilitating the recovery of transmitted messages with minimal error probability.

As shown in Figure 2, only the mapper and demapper modules contain trainable parameters, which are shared between the two. As this system is optimized jointly in an end-to-end manner, the training data are not part of a conventional pre-prepared dataset but are dynamically generated during the training process. At the start of each training iteration, a sufficiently long bit stream is randomly generated. In the mapper, the input random bit stream is first utilized to generate constellation-point symbol labels based on the modulation order (for instance, the 16-QAM modulation labels such as ‘0110’, ‘0010’, etc.). Subsequently, these labels are used to select the corresponding points from the constellation generated by trainable weight parameters, completing the constellation mapping. In the demapper, we implement the demapping by calculating the LLR. The LLR estimates the probability of each transmitted bit being 0 or 1 from the received signal and outputs the LLR. After performing a forward propagation through an end-to-end system, we obtain the communication and sensing performance losses. These losses are combined into a joint loss function. Subsequently, the Adam optimization algorithm is employed to update the trainable weight parameters, modifying the mapping relationship of the real and imaginary parts of the constellation diagram, ultimately achieving constellation optimization.

3.2. Constrained Channel Module

During the neural network training process, PN and PA nonlinearity constraints are explicitly incorporated into the channel model to construct an impairment-augmented-equivalent channel. This drives the network to learn robust constellation diagrams with nonlinear predistortion compensation capabilities.

To depict the nonlinear characteristics of PA, a modified Rapp model is considered, which models the amplitude and phase amplitude distortion as [34,35]

$$G\left(A\right)=gA{\left[1+{\left({\displaystyle \frac{gA}{A_{\mathrm{sat}}}}\right)}^{2s}\right]}^{-{\displaystyle \frac{1}{2s}}},\Psi \left(A\right)=a{A}^{q_1}{\left[1+{\left({\displaystyle \frac{A}{\beta }}\right)}^{q_2}\right]}^{-1},$$

(9)

where A is the input signal amplitude, $A_{\mathrm{sat}}$ is the PA saturation level, s is the smoothness parameter for the amplitude compression curve, and $g, q_1, q_2, \beta$ are parameters, respectively.

The Gaussian PN model is adopted for its analytical tractability in deriving key performance metrics like BER and CRB. Gaussian assumption also simplifies gradient computation for signal design optimization, such as constellation shaping and power allocation. The received signal under PN and AWGN is expressed as $y_k=x_k{e}^{j{\varphi }_k}+w_k$, where $x_k\in \mathbb{C}$ is the transmitted symbol, $\mathbb{E}\left[\right|x_k{|}^2]=P_{\mathrm{avg}}$. ${\varphi }_k\sim \mathcal{N}(0, {\sigma }_{\varphi }^2)$ is the phase noise sample reflecting oscillator stability, and $w_k\sim \mathcal{CN}(0, {\sigma }_n^2)$ is the complex AWGN with ${\sigma }_n^2=N_{0}B$, where B and $N_0$ denote the channel bandwidth and noise power spectral density, respectively.

This approach achieves adaptive constellation mapping tailored for IM-OFDM systems. By integrating channel-state information and nonlinear distortion characteristics into neural network training, the mapping scheme dynamically optimizes symbol distribution based on actual channel conditions. By incorporating PA nonlinear and PN into an equivalent channel model, the neural network is driven to train robust constellation diagrams with nonlinear predistortion compensation capabilities, resulting in spiral-like phase-dimensional distributions that suppress phase rotation interference while ensuring PAPR compliance with system transmission requirements.

3.3. Loss Function

To evaluate the communication performance while considering the impact of symbol confidence on system robustness, cross-entropy is selected as a refined communication performance metric [36], given by

$${\mathcal{I}}_{\mathrm{ce}}=-\sum _{i=1}^M{P}_{\mathrm{true}}\left(i\right)log{P}_{\mathrm{pred}}\left(i\right)$$

(10)

where $P_{\mathrm{true}}$ is the true probability distribution of transmitted symbols and $P_{\mathrm{pred}}$ is the predicted probability distribution from the receiver. The cross-entropy loss penalizes low-confidence correct classifications by assigning higher loss values and encourages output distributions closer to ground truth through gradient updates. Under dynamic channel conditions, this metric effectively quantifies receiver sensitivity to channel impairments.

The core objective of sensing systems lies in high-precision parameter estimation, such as range, velocity, and angle. The CRB, serving as the theoretical performance lower bound, characterizes the minimum mean square error (MSE) of unbiased estimators and evaluates the optimal achievable estimation accuracy. When the estimation parameter is $\mathsf{\theta }=[\tau ,f_D]$, we obtain $\mathrm{CRB}\left(\mathsf{\theta }\right)={\mathbf{J}}^{-1}\left(\mathsf{\theta }\right)$, where $\mathbf{J}\left(\mathsf{\theta }\right)$ denotes the Fisher information matrix (FIM), yielding

$$\begin{array}{c}\hfill \mathbf{J}\left(\mathsf{\theta }\right)={\displaystyle \frac{2}{{\sigma }^2}}\sum _{j=0}^{N_s-1}\sum _{m=0}^{M-1}\left(|{\mathit{X}}_{M\times N_s}{|}^2{\displaystyle \frac{\partial {\mathit{F}}_{\mathrm{Tx}}^{*}}{\partial {\theta }_i}}{\displaystyle \frac{\partial {\mathit{F}}_{\mathrm{Tx}}}{\partial {\theta }_j}}\right),\end{array}$$

(11)

where $F_{\mathrm{Tx}}(j,m)=H_p{e}^{\mathrm{j}2\pi (f_{D}j{T}_s-m\Delta f\tau )}$ is the element of matrix ${\mathbf{F}}_{\mathrm{Tx}}$. $\left\{{\theta }_i,{\theta }_j\right\}\in \theta$ corresponds to ${\theta }_1=\tau$ and ${\theta }_2=f_{\mathrm{D}}$. Then, $\mathbf{J}\left(\mathsf{\theta }\right)$ is a matrix whose elements are as follows:

$$\mathbf{J}\left(\mathsf{\theta }\right)=\left[\begin{array}{cc}\mathrm{J}\left({\theta }_{1,1}\right)\hfill & \mathrm{J}\left({\theta }_{1,2}\right)\hfill \\ \mathrm{J}\left({\theta }_{2,1}\right)\hfill & \mathrm{J}\left({\theta }_{2,2}\right)\hfill \end{array}\right],$$

(12)

where

$$\begin{array}{cc}\hfill \mathrm{J}\left({\theta }_{1,1}\right)& ={\displaystyle \frac{2}{{\sigma }^2}}\sum _{j=0}^{N_s-1}\sum _{m=0}^{M-1}{\left|x_m\left(j\right)\right|}^2{\left(2\pi m\Delta f\right)}^2,\hfill \\ \hfill \mathrm{J}\left({\theta }_{1,2}\right)& =J\left({\theta }_{2,1}\right)={\displaystyle \frac{2}{{\sigma }^2}}\sum _{j=0}^{N_s-1}\sum _{m=0}^{M-1}-{\left|x_m\left(j\right)\right|}^2\left(4\pi jm{T}_s\Delta f\right),\hfill \\ \hfill \mathrm{J}\left({\theta }_{2,2}\right)& ={\displaystyle \frac{2}{{\sigma }^2}}\sum _{j=0}^{N_s-1}\sum _{m=0}^{M-1}{\left|x_m\left(j\right)\right|}^2{\left(2\pi j{T}_s\right)}^2.\hfill \end{array}$$

(13)

Then, the CRB function of parameters $\tau$ and $f_{\mathrm{D}}$ is given by

$$\begin{array}{c}\hfill \mathrm{CRB}\left(\tau \right)={\displaystyle \frac{\mathrm{J}\left({\theta }_{2,2}\right)}{\mathrm{J}\left({\theta }_{1,1}\right)\mathrm{J}\left({\theta }_{2,2}\right)-J\left({\theta }_{1,2}\right)\mathrm{J}\left({\theta }_{2,1}\right)}},\\ \hfill \mathrm{CRB}\left(f_D\right)={\displaystyle \frac{\mathrm{J}\left({\theta }_{1,1}\right)}{\mathrm{J}\left({\theta }_{1,1}\right)\mathrm{J}\left({\theta }_{2,2}\right)-\mathrm{J}\left({\theta }_{1,2}\right)J\left({\theta }_{2,1}\right)}}.\end{array}$$

(14)

To achieve a dynamic balance between communication and sensing performance, a weighted joint loss function $Loss$ is constructed. For the joint performance of the communication–sensing system, the weighting factor $\alpha$ is employed to combine communication performance and sensing performance, yielding

$$\begin{array}{c}\hfill Loss=(1-\alpha )\underset{\mathrm{Communication}}{\underbrace{{\mathcal{L}}_{\mathrm{CS}}\left({\mathcal{I}}_{\mathrm{ce}};\mathit{c}\right)}}+\alpha \underset{\mathrm{Sensing}}{\underbrace{{\mathcal{L}}_{\mathrm{SC}}\left(\mathrm{CRB}\left(\mathsf{\theta }\right);\mathit{c}\right)}},\end{array}$$

(15)

where the weighting factor $\alpha \in [0, 1]$ controls the priority between communication and sensing objectives, enabling adaptive constellation design for different scenarios. In communication-priority mode ($\alpha \to 0$), high-density irregular constellation mappings are adopted to maximize BER. In sensing-priority mode ($\alpha \to 1$), regular sparse constellation mappings are designed to minimize CRB. The weighting factor $\alpha$ can be dynamically adjusted based on real-time channel conditions, i.e., increase $\alpha$ to prioritize ranging accuracy in high-Doppler-shift scenarios and reduce $\alpha$ to optimize the system in low-SNR environments. Then, the framework essentially finds the optimal constellation ${\mathit{c}}^{*}=arg{min}_{\mathcal{C}}Loss$, where $\mathcal{C}$ denotes all possible constellation mappings.

3.4. Optimal Training Method Based on Adam

Considering that gradient-directed search is more suitable for constellation optimization in terms of complexity and real-time performance, we select an appropriate gradient descent algorithm. For constellation optimization design, the real and imaginary parts of constellation points can be set as network weights to optimize the constellation through loss minimization. When the loss function represents system performance, this essentially identifies the constellation that maximizes communication–sensing system performance.

Stochastic Gradient Descent (SGD) is a gradient-based optimization algorithm that minimizes loss functions by updating parameters using gradients computed from randomly sampled data batches. Define the constellation mapping function as ${\mathit{f}}_{\mathit{c}}\in \mathcal{F}$, where $\mathcal{F}:{\mathit{b}}_B\left(n\right)\to \mathit{c}\left(n\right)$ is the set of all possible constellation mapping relationships. For constellation diagram optimization, the real and imaginary parts of constellation points $\Re \left(c\right(i\left)\right), \Im \left(c\right(i\left)\right)$ are directly parameterized as network weights, where $\mathit{W}=\{\Re \left(c\left(1\right)\right), \Im \left(c\left(1\right)\right),\dots , \Re \left(c\left(N_c\right)\right), \Im \left(c\left(N_c\right)\right)\}$, and M is the modulation order.

Compared to SGD, which only considers the first moment estimate and is prone to becoming trapped in local optima, Adam combines momentum gradients with adaptive learning rates, taking into account both first- and second-moment estimates. This allows for adaptive adjustment of the learning rate, enabling faster escape from local optima, resulting in more accurate and efficient convergence. The process is depicted as Algorithm 1. The parameter settings for Adam include the learning rate, exponential decay rate (EDR) for the first-moment estimates, EDR for the second-moment estimates, and a small constant for numerical stability.

Algorithm 1 End-to-End Constellation Optimization Algorithm Based on Adam

Require: Random bitstream ${\mathit{b}}_B$, The set of network weights of neural network ${\mathit{W}}_0$.   1: Initial ${\mathit{W}}_0$, Adam learning rate = 0.001, 1st EDR = 0.9, 2st EDR = 0.999.   2: for epoch $=1$ to $N_{\max }$ do   3:     Forward pass: $\widehat{b}=\mathrm{Decoder}(H·{\mathit{f}}_{\mathit{c}}\left(W\right)+n)$   4:     Compute ${\mathcal{I}}_{\mathrm{ce}},\mathrm{CRB}\left(\theta \right)$   5:     Evaluate joint loss:   6:     $Loss=(1-\alpha ){\mathcal{L}}_{\mathrm{CS}}({\mathcal{I}}_{\mathrm{ce}};{\mathit{f}}_{\mathit{c}})+\alpha {\mathcal{L}}_{\mathrm{SC}}\left(\mathrm{CRB}\right(\theta );{\mathit{f}}_{\mathit{c}})$   7:     Backpropagate gradients: ${\nabla }_{W}Loss\leftarrow \mathrm{Backprop}\left(Loss\right)$   8:     Adam update ${\mathit{W}}_{epoch+1}\leftarrow {\mathit{W}}_{epoch}$.   9: end for

4. Experiments

In this paper, we present experiments that were conducted on the proposed CO-IM-OFDM system, comparing it with existing traditional constellation mappings, including 16APSK, 16spiral, and 16QAM. The training employed the Keras deep learning framework for network model construction, with the open-source library Sionna serving as the backend engine for link-level simulations. We also investigated the impact of different loss function weighting factors on the proposed scheme. The simulation configurations for the neural network-empowered constellation optimization design are shown in

Table 1. Simulation parameters for constellation optimization.

Parameter	Configuration
Neural network architecture	$2\times 16$
Parameter initialization	$\Re \left(c\right(i\left)\right),\Im \left(c\right(i\left)\right)$ of 16QAM
Training data	Random binary bit stream
Optimizer	Adam optimizer
Learning rate	0.001
EDR for first-moment estimates	0.9
EDR for second-moment estimates	0.999
Small constant for numerical stability	$1\times {10}^{-7}$
Loss function	$Loss$
Batch size	64
Training epochs	30,000

.

Considering the CO-IM-OFDM system, index modulation corresponds to $N_g=4$ and $k=2$, and the other simulation parameters for CO-IM-OFDM are shown in

Table 2. Simulation parameter configuration for CO-IM-OFDM.

Simulation Parameter	Setting
FFT size	128
OFDM symbol count	64
PA model	Rapp model ($P_{\mathrm{aSat}}=0.58$)
PN variance	0.01, 0.1
Channel model	Rayleigh channel (path = 1, 10, 20)
Training $E_b/N_0$ range	10–20 dB

.

Figure 3 demonstrates the training results under different weighting factors. It can be observed that when communication performance is optimized ($\alpha \to 0$), the constellation exhibits a spiral-like structure to counteract signal-rotation-induced PN. Simultaneously, the inner and outermost constellation points maintain sufficient separation to preserve Euclidean distance uniformity under PA nonlinear distortion. Conversely, when sensing performance is prioritized ($\alpha \to 1$), the constellation approximates a constant-envelope structure where the signal’s constant modulus property ensures stable sensing capabilities. However, this sensing-optimal configuration sacrifices amplitude diversity, a critical degree of freedom for information encoding, leading to degraded communication performance. This demonstrates an inherent deterministic–stochastic trade-off between communication and sensing objectives: communication-driven design requires maximized signal randomness to enhance information capacity, and sensing-driven design demands deterministic signal features to guarantee sensing reliability.

Figure 4 illustrates the results of balancing communication and sensing performance through weighting factor adjustment. The allocation of a weighting factor essentially represents a resource allocation trade-off, which is adjusted according to specific application scenarios. Through varying the weighting factor, we establish a constellation design framework and obtain the joint CRB-BER boundary for IM-OFDM under constellation optimization. Curve A represents the ideal state of the ISAC system, where both communication and sensing maintain optimal performance without mutual interference. However, this state is known to be unattainable. Curve B depicts the current baseline ISAC system state, which compromises partial communication performance to achieve sensing. This approach provides no additional performance gain and consequently causes performance degradation in the other objectives. The closed region formed by Curves A and B represents the achievable integrated performance area, with the research goal being to approach the performance upper limit (Curve A). Curve C shows performance obtained through continuous weighting factor adjustment and the proposed end-to-end training. It can be observed that BER increases with $\alpha$, while CRB decreases with $\alpha$. The trained constellation diagram demonstrates performance gains compared to Curve B, approaching the theoretical upper limit.

In practical scenarios, selecting an appropriate $\alpha$ value yields a constellation tailored to the dominant system requirement. In detail, for applications predominantly considering sensing performance, such as autonomous driving, $\alpha >0.5$ would be set; for prioritizing communication performance, such as end-to-end video transmission, $\alpha <0.5$ would be set. This resource allocation problem depends heavily on the specific application context. We acknowledge that a more detailed investigation into optimal $\alpha$ selection strategies would be a valuable point of focus for future research.

Then, we provide simulation results under Gaussian channels and Rayleigh fading channels with different numbers of paths. By varying the number of multipath components as 1, 10, 20, and PN level as 0.01, 0.1, we simulated in different channel conditions.

As shown in Figure 5, we simulated the CRB-BER joint performance under Rayleigh and AWGN channel conditions. It can be observed that all simulation curves exhibit a concave characteristic. This phenomenon indicates that in the trade-off between BER and CRB, the system achieves a significant improvement in communication reliability by sacrificing a small amount of sensing accuracy, thus obtaining performance gains. This concave trend is consistently present across all simulated channel conditions. In addition, as PN increases, the system performance deteriorates, with both endpoints of the curve, optimal communication and optimal sensing points, shifting back. This fully demonstrates that the proposed neural network architecture can adapt to different channel conditions and effectively achieve performance optimization in joint communication and sensing design.

Next, the performance indicators, including BER, throughput, and CRB under various weight factors, as well as their variations with respect to SNR, are compared. The benchmark for comparison is based on traditional 16QAM modulation.

Figure 6 illustrates the BER performance curves under varying $E_b/N_0$ and weight factors. For the traditional 16QAM baseline, the BER decreases slowly at high SNR, remaining as high as ${10}^{-1}$ at $E_b/N_0=20\mathrm{dB}$. With the proposed algorithm, performance improves significantly as $\alpha$ decreases. Specifically, when communication performance ($\alpha =0$) is prioritized, BER outperforms the baseline by approximately 10 dB under the same conditions, exhibiting a steep decline in high-SNR regions. Conversely, $\alpha =1$ yields the worst BER due to the enforced constant-envelope constraint, demonstrating the limitation of deterministic signals on communication reliability.

Figure 7 shows the CRB curves under varying $E_b/N_0$ and weight factors. As SNR increases, the CRB decreases. When prioritizing communication performance, the proposed algorithm slightly underperforms compared to 16QAM. However, in sensing-priority mode ($\alpha =0.2\sim 1$), the CRB decreases by 2∼8% as the weight factor increases, indicating optimized parameter estimation accuracy through deterministic signal structures.

Next, the performance of the constellation adapted for IM-OFDM via neural networks is compared with the existing constellations. We select 16APSK, 16QAM, and 16spiral as algorithms for comparison. Their constellation diagrams are shown in Figure 8. Among them, the amplitude of 16APSK satisfies the constant envelope characteristic, enabling it to combat nonlinear channels and making it suitable for power-constrained scenarios. The constellation points of 16QAM are evenly distributed, offering high-frequency utilization, but it cannot effectively resist nonlinear distortion. The 16spiral constellation exhibits a spiral structure, possessing excellent PN resistance properties. These constellation diagrams cannot simultaneously satisfy the requirements of resisting PN, PA nonlinearity, and ensuring communication and sensing performance.

Figure 9 illustrates BER performance curves of the proposed constellation and classical constellations when applied to IM-OFDM systems. When sensing performance is prioritized ($\alpha =1$), the proposed scheme exhibits degraded BER performance, even underperforming 16QAM and 16APSK. However, for $0\le \alpha \le 0.8$, the proposed constellation achieves remarkable BER improvements. For instance, at $\alpha =0.8$, it attains a 5 dB gain over 16QAM under the same BER conditions, demonstrating its adaptability to channel constraints and system requirements. Furthermore, smaller $\alpha$ values correlate with better BER performance, with optimization effects becoming more pronounced in high-SNR regions ($E_b/N_0>18\mathrm{dB}$). These results confirm that the proposed algorithm dynamically balances signal randomness and deterministic structures to enhance demodulation robustness.

Figure 10 illustrates throughput performance for different training results. The simulation results indicate that, in terms of throughput performance, all the values of $\alpha$, except for 1, significantly outperformed the traditional 16QAM and 16APSK. It is noteworthy that the result for $\alpha =1$ showed a decrease in throughput compared to the baseline scheme. Under full sensing conditions, the training process solely maximizes the sensing performance without considering the communication performance. Consequently, the network parameters under this mode cannot effectively support efficient data transmission, leading to lower communication throughput than the baseline scheme that balances both communication and sensing. The significant improvement in throughput performance observed at other values of $\alpha$ fully demonstrates that the proposed joint optimization architecture can effectively achieve performance gains in terms of throughput.

Figure 11 depicts CRB performance curves of the proposed constellation and benchmark constellations in IM-OFDM systems. When communication performance is prioritized ($\alpha =0$), the CRB degrades significantly due to the signal’s high randomness, which introduces amplitude fluctuations detrimental to parameter estimation accuracy. However, as $\alpha$ increases within $0.2\sim 1$, the CRB decreases, reaching its minimum at $\alpha =1$. This trend aligns with the theoretical trade-off between deterministic signal structures and stochastic modulation diversity in ISAC systems. The joint analysis of BER and CRB performance reveals that, except for the communication-optimal ($\alpha =0$) and sensing-optimal ($\alpha =1$) configurations, all trained constellations achieve significant performance gains compared to conventional 16QAM, 16APSK, and 16spiral schemes.

5. Conclusions

In this paper, we proposed a CO-IM-OFDM framework that integrates neural network-trained constellation mappings with index modulation techniques. We designed optimized constellations adaptive to dynamic subcarrier activation patterns in IM-OFDM systems, aiming to maximize spectral efficiency while adhering to PAPR and PN constraints. Furthermore, the communication–sensing joint performance was defined as the primary optimization objective, where the Adam optimizer is employed to train the constellation design. Simulation results demonstrated that the proposed method achieves BER performance gains exceeding 4 dB and CRB improvements of 2–8%, substantially outperforming conventional fixed-constellation IM-OFDM systems in terms of both communication and sensing capabilities.