Rate–Distortion–Perception Optimized Neural Speech Transmission System for High-Fidelity Semantic Communications ^†

Abstract

We consider the problem of learned speech transmission. Existing methods have exploited joint source–channel coding (JSCC) to encode speech directly to transmitted symbols to improve the robustness over noisy channels. However, the fundamental limit of these methods is the failure of identification of content diversity across speech frames, leading to inefficient transmission. In this paper, we propose a novel neural speech transmission framework named NST. It can be optimized for superior rate–distortion–perception (RDP) performance toward the goal of high-fidelity semantic communication. Particularly, a learned entropy model assesses latent speech features to quantify the semantic content complexity, which facilitates the adaptive transmission rate allocation. NST enables a seamless integration of the source content with channel state information through variable-length joint source–channel coding, which maximizes the coding gain. Furthermore, we present a streaming variant of NST, which adopts causal coding based on sliding windows. Experimental results verify that NST outperforms existing speech transmission methods including separation-based and JSCC solutions in terms of RDP performance. Streaming NST achieves low-latency transmission with a slight quality degradation, which is tailored for real-time speech communication.

1. Introduction

The vast demand of streaming audio and video communication poses significant challenges to wireless communication systems, underscoring the need to elevate both the quality and efficiency of speech transmission. Current wireless communication systems suffers from the cliff effect where the signal reconstruction quality breaks down if the channel quality falls below the level anticipated by the channel code. Learning-based speech transmission methods [1,2,3] are emerging as promising solutions to improve the end-to-end transmission performance in the context of semantic communication [4,5,6,7]. They mostly leverage the idea of joint source–channel coding (JSCC) to produce transmitted symbols directly from raw speech signals with neural networks, which is featured with graceful degradation with respect to channel quality [1,2,8]. However, these approaches fail to identify the content diversity among signals, leading to inefficient transmission. Streaming inference is also a fundamental aspect in real-time communication (RTC) scenarios. Although transmission errors can be compensated by retransmission such as hybrid automatic repeat requests, these lead to a loss of efficiency and transmission delay.

To address the above-mentioned issues, we make the first attempt to design a high-fidelity neural speech transmission framework (NST) for better end-to-end transmission performance. Motivated by learned data compression techniques [9,10], NST establishes a learned entropy model on latent speech features and then realizes semantic-guided variable-length joint source–channel coding, thus achieving better coding gain. Specifically, a critical set of hyperprior variables is established upon the latent features, which estimate the entropy of speech features by variational modeling. Under the guidance, speech latent features are dynamically encoded to variable-length symbol sequences via a joint source–channel encoder. Based on our previous work [11], we further investigate the real-time speech transmission within latency-sensitive contexts, such as online conferencing and voice calls. In particular, we develop a streaming variant of NST tailored for low-latency transmission. All the operators of the model are strictly causal ones, which attend to the past speech signals only, to satisfy the real-time property. In addition, we design a sliding-window based inference mechanism in joint source–channel coding, which balances the performance of speech reconstruction and the overall delay.

We evaluate the performance by conducting simulations over wireless channels. The results demonstrate that the proposed NST model is source and channel-adaptive. In comparison to advanced speech coding combined with error correction coding, and the existing JSCC solution, the proposed NST achieves a superior rate–distortion–perception tradeoff. This translates to a high-fidelity speech reconstruction performance while incurring lower bandwidth costs. Notably, the streaming NST makes a slight compromise in speech quality to meet the low-latency requirement.

Notational Conventions: Throughout this paper, bold letters (e.g., $\mathit{x}$) denote vectors and the scalars, and lowercase ones denote scales. Bold uppercase letters (e.g., $\mathcal{V}$) represent a collection. $log\left(·\right)$ is the logarithm to base 2. $p_x$ denotes a probability density function (pdf) with respect to the continuous-valued random variable x. $\mathcal{U}(a-m,a+m)$ denotes a uniform distribution centered on a with width $2m$. $\mathbb{R}$ and $\mathbb{C}$ denote the real number set and the complex number set, respectively. $\mathbb{E}\left[·\right]$ denotes the statistical expectation operation.

2. Methodology

2.1. Architecture

The NST system architecture is illustrated in Figure 1. Assuming a sequence of T speech frames $\mathit{x}=\left\{{\mathit{x}}_1,{\mathit{x}}_2,\cdots {\mathit{x}}_T\right\}$, the analysis transform module $g_a\left(·;{\mathit{\varphi }}_g\right)$, which consists of convolutional neural networks (CNNs) with temporal downsampling, transforms them into a semantic latent feature sequence $\mathit{y}=\left\{{\mathit{y}}_1,{\mathit{y}}_2,\cdots {\mathit{y}}_T\right\}$. Then, the latent features $\mathit{y}$ are fed into both a hyperprior encoder $h_a\left(·;{\mathit{\varphi }}_h\right)$ and a variable-length JSCC encoder $f_e\left(·;{\mathit{\varphi }}_f\right)$. On one hand, in order to conveniently quantify the amount of information for speech features, each element of $\mathit{y}$ is variationally modeled by a simple Gaussian, whose parameters are encapsulated by the hyperprior variable $\mathit{z}$. The means and variances of the Gaussians are encoded by $h_a\left(·;{\mathit{\varphi }}_h\right)$ and $h_s\left(·;{\mathit{\theta }}_h\right)$ to capture the dependencies of $\mathit{y}$. On the other hand, $f_e\left(·;{\mathit{\varphi }}_f\right)$ encodes $\mathit{y}$ into channel-input sequence $\mathit{s}=\left\{{\mathit{s}}_1,{\mathit{s}}_2,\cdots {\mathit{s}}_T\right\}$, where ${\mathit{s}}_i\in {\mathbb{C}}^{k_i}$ is a $k_i$-dimensional complex vector to transmit $y_i$. We consider a wireless channel denoted by $W\left(·;\mathit{\nu }\right)$, where $\mathit{\nu }$ denotes the channel parameters. Thus, the receiver obtains the sequence $\widehat{\mathit{s}}=W\left(\mathit{s};\mathit{\nu }\right)$ with the transition probability $p_{\widehat{\mathit{s}}|\mathit{s}}\left(\widehat{\mathit{s}}|\mathit{s}\right)$. As illustrated in Figure 1, with a mirrored design, the JSCC decoder $f_d\left(·;{\mathit{\theta }}_f\right)$ reconstructs latent representation $\widehat{\mathit{y}}$, and semantic synthesis transform $g_s\left(·;{\mathit{\theta }}_g\right)$ recovers speech waveform $\widehat{\mathit{x}}$. Hence, the total link of NST is formulated by

$$\mathit{x}\stackrel{g_a\left(·;{\mathit{\varphi }}_g\right)}{\to }\mathit{y}\stackrel{f_e\left(·;{\mathit{\varphi }}_f\right)}{\to }\mathit{s}\stackrel{W\left(·;\mathit{\nu }\right)}{\to }\widehat{\mathit{s}}\stackrel{f_d\left(·;{\mathit{\theta }}_f\right)}{\to }\widehat{\mathit{y}}\stackrel{g_s\left(·;{\mathit{\theta }}_g\right)}{\to }\widehat{\mathit{x}},$$

(1)

with the latent prior $\mathit{y}\stackrel{h_a\left(·;{\mathit{\varphi }}_h\right)}{\to }\mathit{z}\stackrel{h_s\left(·{\mathit{\theta }}_h\right)}{\to }\left\{\mathit{\mu },\mathit{\sigma }\right\}$ and $(\mathit{\theta },\mathit{\varphi })=({\mathit{\varphi }}_g,{\mathit{\varphi }}_h,{\mathit{\varphi }}_f,{\mathit{\theta }}_g,{\mathit{\theta }}_h,{\mathit{\theta }}_f)$ encapsulating the learnable parameters of each function above. Moreover, the hyperprior $\mathit{z}$ can be viewed as side information, which is optionally sent via a digital link to the receiver to refine the latent feature $\mathit{y}$.

2.2. Dynamic Variable-Length Joint Source–Channel Coding

As defined previously, each $y_i$ is variationally modeled as a Gaussian with mean ${\mu }_i$ and variance ${\sigma }_i^2$, whose density function is factorized as

$$p\left(\mathit{y}|\mathit{z};{\mathit{\theta }}_h,{\mathit{\psi }}_h\right)=\prod _i\underset{p_{y_i|\mathit{z}}}{\underbrace{\left(\mathcal{N}\left({\mu }_i,{\sigma }_i^2\right)\ast \mathcal{U}\left(-\frac{1}{2},+\frac{1}{2}\right)\right)}}\left(y_i\right),$$

(2)

with $(\mathit{\mu },\mathit{\sigma })=h_s\left(\mathit{z}\right)$, where ∗ is a convolutional operation. Dithered quantization is adopted [12], such that we can derive a non-negative entropy estimation of $-log{p}_{\mathit{y}|\mathit{z}}\left(\mathit{y}\right|\mathit{z})$ by directly using the proxy ${\tilde{y}}_i=y_i+o,o\in \mathcal{U}\left(-\frac{1}{2},+\frac{1}{2}\right)$. The estimated entropy is directly linked to the channel bandwidth cost in the JSCC encoder for transmission. Intuitively, if ${\mathit{y}}_i$ is tagged with high entropy, it will be allocated with more bandwidth and vice versa.

In practice, the total bandwidth cost $K_{\mathit{y}}$ for transmitting $\mathit{y}$ is formulated by

$$K_{\mathit{y}}=\sum _{i=1}^T{\overline{k}}_{y_i}=\sum _{i=1}^{T}Q\left(k_{y_i}\right)=\sum _{i=1}^{T}Q(-{\eta }_{y}log{p}_{{\mathit{y}}_i|\mathit{z}}\left({\mathit{y}}_i\right|\mathit{z})),$$

(3)

where ${\eta }_y$ controls the scaling between the estimated entropy and the number of transmitted symbols, and Q denotes a $2^n$-level scalar quantization with the quantized value set as $\mathcal{V}=\{v_1,v_2,\cdots ,v_{2^n}\}$. Hence, n bits are transmitted as side information to inform the receiver in which ${\overline{k}}_{y_i}\in \mathcal{V}$ is selected for transmitting ${\mathit{y}}_i$.

We adopt a pair consisting of a Transformer-like [13] JSCC encoder and decoder as $f_e$ and $f_d$, as plotted in Figure 2. Guided by the entropy model $-log{p}_{\mathit{y}|\mathit{z}}\left(\mathit{y}\right|\mathit{z})$, a set of learnable rate token embeddings with the same dimension with $y_i$ are developed, each of which corresponds to a value in $\mathcal{V}$. To adapt to various channel environments, we assume a channel state information feedback to inform the sender of the instant signal-to-noise ratio (SNR). Similarly, a set of learnable SNR tokens are developed. T frames of speech features are gathered and fused with respective rate tokens and an SNR token, and they are finally fed into the Transformer block with $N_e$ Transformer layers. A bunch of fully connected (FC) layers with output dimensions of $v_q,q=1,2,\cdots ,2^n$ are employed to map the embeddings into $s_i$ with given dimensions. A toy visualization of the rate allocation result is displayed in Figure 3. It can be observed that more bandwidth is allocated to frames with prominent contents and less is allocated to ones in silence. The overall bandwidth is adjusted by tuning the hyperparameter ${\eta }_y$.

2.3. Streaming NST for Real-Time Communication

In this subsection, we propose a streaming NST model variant to facilitate real-time speech communication.

Firstly, all the convolutional operators in $g_a$ and $g_s$ are substituted by causal ones. The transposed convolution operations in $g_s$ only pad on the past steps to meet the causal property. Secondly, the joint source–channel coding of speech latent features is modified from that in Figure 3 to reduce the latency. Traditionally, Transformer uses multi-head attention that jointly learns diverse relationships between queries and keys, which are the speech features $\mathit{y}$ in this paper, from different representation subspaces with j-th head computing

$${\mathit{Q}}_j={\mathit{W}}_j^Q\mathit{y},{\mathit{K}}_j={\mathit{W}}_j^K\mathit{y},{\mathit{V}}_j={\mathit{W}}_j^V\mathit{y}.$$

(4)

To meet the real-time requirement, a causal masked attention method is proposed together with a sliding-window inference mechanism. As shown in Figure 4, the JSCC encoder $f_e$ has a limited contextual window with W frames, which hops along the temporal domain with a stride of N frames. In particular, we follow Transformer-XL [14] and create a segment-level recurrence of output by intermediate layers. In this paper, we define the attention span of each layer as $2N$ frames, which ends with the last frame of the current N target frames. The self-attention is computed after a causal mask $\mathit{M}$ is applied, whose elements satisfy

$${\mathit{M}}_{t,\tau }=\left\{\begin{array}{cc}1,\hfill & t-2N<\tau \le t\hfill \\ -\infty ,\hfill & \mathrm{others}\hfill \end{array}\right..$$

(5)

Then, the output of the j-th head self-attention ${\mathit{a}}_j$ is formulated by

$${\mathit{a}}_j=\mathrm{Softmax}\left(\frac{{\mathit{Q}}_j{\mathit{K}}_j^{\mathrm{T}}}{\sqrt{d_h}}\odot \mathit{M}\right){\mathit{V}}_j,$$

(6)

where $d_h$ is the dimension of each head. Thus, the length of contextual window W grows linearly with respect to the number of Transformer layers as well as the window stride, which can be written as $W=N(N_e+1)$ frames.

2.4. Optimization Goal

The analysis transform together with the joint source–channel encoder creates a parametric density $q_{\widehat{\mathit{s}},\tilde{\mathit{z}}|\mathit{x}}$ to approximate the true posterior distribution $p_{\widehat{\mathit{s}},\tilde{\mathit{z}}|\mathit{x}}$. The optimization goal is to minimize the Kullback–Leibler (KL) divergence between the above two components. After reformulation, it minimizes its upper bound, i.e.,

$$min\underset{\mathit{x}\sim p_{\mathit{x}}}{\mathbb{E}}\underset{\widehat{\mathit{s}},\tilde{\mathit{z}}\sim q_{\widehat{\mathit{s}},\tilde{\mathit{z}}}}{\mathbb{E}}{D}_{\mathrm{KL}}\left[q_{\widehat{\mathit{s}},\tilde{\mathit{z}}|\mathit{x}}\parallel p_{\widehat{\mathit{s}},\tilde{\mathit{z}}|\mathit{x}}\right]\le min\underset{\mathit{x}\sim p_{\mathit{x}}}{\mathbb{E}}\underset{\widehat{\mathit{s}},\tilde{\mathit{z}}\sim q_{\widehat{\mathit{s}},\tilde{\mathit{z}}}}{\mathbb{E}}\left[\underset{\mathrm{side}\mathrm{info}.\mathrm{coding}\mathrm{rate}}{\underbrace{-log{p}_{\tilde{\mathit{z}}}\left(\tilde{\mathit{z}}\right)}}\underset{\mathrm{bandwidth}}{\underbrace{-log{p}_{\widehat{\mathit{s}}|\tilde{\mathit{z}}}\left(\widehat{\mathit{s}}\right|\tilde{\mathit{z}})}}\underset{\mathrm{distortion}}{\underbrace{-{\mathbb{E}}_{\mathit{y}\sim p_{\mathit{y}|\widehat{\mathit{s}},\tilde{\mathit{z}}}}log{p}_{\mathit{x}|\mathit{y}}\left(\mathit{x}\right|\mathit{y})}}\right]+\mathrm{const}.$$

(7)

The first term of (7) represents the cost of encoding the side information assuming $p_{\tilde{\mathit{z}}}$ as the entropy model, where ${\tilde{z}}_i=z_i+o$ is the proxy quantization of $z_i$. Since there is no prior information about $\mathit{z}$, $p_{\tilde{\mathit{z}}}\left(\tilde{\mathit{z}}\right)$ is modeled as a non-parametric fully factorized density [9]$p_{\tilde{\mathit{z}}}\left(\tilde{\mathit{z}}\right)={\prod }_i\left(p_{z_i|{\mathit{\psi }}^{\left(i\right)}}\left(z_i\right|{\mathit{\psi }}^{\left(i\right)})\ast \mathcal{U}(-\frac{1}{2},\frac{1}{2})\right)\left(z_i\right)$. The second term represents the bandwidth cost of encoding $\widehat{\mathit{s}}$. In practice, the intermediate variable $\mathit{y}$ is utilized by $p_{\widehat{\mathit{s}}|\tilde{\mathit{z}}}=W\left(p_{\mathit{s}|\tilde{\mathit{z}}}\right|\mathit{h})=W\left(f_e\left(p_{\mathit{y}|\tilde{\mathit{z}}}\right)\right|\mathit{h})$. The third term denotes the weighted distortion of the reconstructed speech waveform. $d(·,·)$ indicates the objective signal distortion. To enrich the distortion term in alignment with human perceptual quality, a differentiable $F(·)$ is employed as a perceptual feature extractor, and the distance between perceptual features $d_p(·,·)$ is minimized to improve the listening quality.

In summary, the RDP function is formulated as

$$L_{\mathrm{RDP}}(\mathit{\theta },\mathit{\varphi },\mathit{\psi })={\mathbb{E}}_{\mathit{x}\sim p_{\mathit{x}}}[-{\eta }_{y}log{p}_{\mathit{y}|\mathit{z}}\left(\mathit{y}|\mathit{z}\right)-{\eta }_{z}log{p}_{\tilde{\mathit{z}}}\left(\tilde{\mathit{z}}\right)+{\lambda }_{D}d(\mathit{x},\widehat{\mathit{x}})+{\lambda }_P{d}_p(F_{\mathit{x}},F_{\widehat{\mathit{x}}})],$$

(8)

where the Lagrange multipliers ${\lambda }_D,{\lambda }_P$ control the tradeoff among the total transmission rate, the distortion and the perceptual quality. The scaling factor ${\eta }_y$ is adjusted for RDP tradeoff, while ${\eta }_z$ is determined according to the channel capacity of the optional tranmission link.

3. Results

In this section, we provide numerical results in terms of objective quality metrics and subjective scores to evaluate the quality of speech transmission.

3.1. Experimental Setup

The mono speech signals are sampled at 16 kHz from the TIMIT dataset [15]. Compared to our conference paper [11], to adapt to RTC scenarios, a shorter frame length is considered in this paper. Each speech frame has $L=128$ samples with an overlap of eight samples. The analysis transform module $g_a$ and synthesis transform module $g_s$ consist of stacks of 1D convolutional layers with a residual connection. The number of channels of the convolutional kernel of the output/input layer for $g_a$/$g_s$ is configured with $C_g=4$, while the one for $h_a$/$h_s$ is set as $C_h=2$. In the variable-length JSCC coder $f_e$ and $f_d$, we use $N_e=3$ Transformer layers with eight-head self-attention. The quantized channel bandwidth cost value set is defined as $\mathcal{V}=\left\{10,40,90,120,200,250,300,400\right\}$. Each speech frame ${\mathit{x}}_i\in {\mathbb{R}}^{1\times L}$ is transformed into latent feature ${\mathit{y}}_i\in {\mathbb{R}}^{C_g\times \frac{L}{4}}$ with a downsampling factor of four. It is then flattened into an embedding vector with a dimension of $\frac{C_{g}L}{4}=128$, which is identical to the dimension of the Transformer in JSCC coders.

In (8), the object signal distortion d is evaluated by the mean square error in the time domain. In terms of perceptual optimization, we minimize the difference of Mel-frequency cepstral coefficients (MFCCs) [16], which is a hand-crafted speech perceptual feature. Specifically, a mean square loss function $d_p$ for MFCCs is employed, where F denotes the function of the MFCC extractor.

We compare our NST model with traditional separation-based transmission schemes. Specifically, we employ the widely used speech codec AMR-WB [17] and Opus [18] for source coding and convolutional codes, 5G LDPC [19] for channel coding, and follow the principle of adaptive modulation coding (AMC) [20]. Moreover, we also compare our NST model with another JSCC model DeepSC-S [1] for speech transmission, which is a non-streaming model with CNN modules. We modify its model to support low bandwidth transmission with 12 kHz and 32 kHz, separately.

3.2. Evaluation Metrics

In terms of objective metrics for perceptual quality, we report the perceptual evaluation of speech quality (PESQ) [21] scores, which range from 1.0 to 4.5. Furthermore, we implement a Multiple Stimuli with Hidden Reference and Anchor (MUSHRA) subjective test [22] for human preference evaluation. As a widely used approach in the subject quality assessment method, the MUSHRA test allows users to compare multiple variants of reconstructed audio and provides the relative score between 0 and 100. We randomly select 10 speech segments from the test set.

3.3. Results Analysis

Figure 5 reports the PESQ performance over additive white Gaussian noise (AWGN) channels. In Figure 5a, with a fixed channel bandwidth cost of $K=$ 10 kHz, we find that the proposed NST brings a performance gain for all SNRs by incorporating source and channel information into JSCC, especially in a low SNR region. In addition to traditional speech source coding methods, we also compare with a nonlinear neural speech compressor which employs the similar entropy model as NST to entropy encode the latent speech features. This scheme is marked in the figure as “NTC + QPSK, 1/2 Conv” when using convolutional codes with a rate of 1/2 and QPSK modulation as “NTC + 5G LDPC” when using LDPC codes. NST demonstrates graceful performance degradation with the decrease of SNR, while the performance of other separation-based methods probably breaks down (cliff effect) when using a single-channel coding rate and modulation level, e.g., “NTC + QPSK, 1/2 Conv”. For the 5G LDPC, we plot the envelope of several curves, corresponding to different coding rates and modulation levels. Compared with another JSCC method DeepSC-S, our model achieves better perceptual quality by introducing an explicit perceptual loss function with much less bandwidth cost. In addition, we notice a slight quality drop under the streaming inference setting, but it remains better than other methods. NST adapts well to various channel conditions by means of SNR token fusion in $f_e$ and $f_d$ using a single model.

Figure 5b compares the rate–distortion–perception performance using different methods for the 6 dB AWGN channel. Since the NST model learns an adaptive rate allocation mechanism, we traverse the ${\eta }_y$ from 0.1 to 0.3 and finetune the model with a fixed ${\lambda }_D$ and ${\lambda }_P$. It can be observed that a remarkable bandwidth saving can be accomplished for NST by integrating source semantic information as well as channel information.

To delineate the distortion–perception tradeoff, we conduct an ablation study examining the impact of perceptual optimization. We evaluate the signal–to–distortion ratio (SDR) performances to assess the traditional signal distortion. The results in Figure 6 demonstrate that the proposed NST using an RDP optimization objective function (8) outperforms its counterpart solely optimized toward reduced signal distortion in terms of perceptual quality. NST with rate–distortion (RD) optimization (omitting perceptual loss in (8)) exhibits inferior perceptual quality despite there being less objective signal distortion. Performance using the traditional speech coding method is also included in the figure, which also underscores the significance of perceptual optimization in addition to minimizing the objective signal distortion.

Figure 7 displays the effect of SNR fusion in our SNR-adaptive joint source–channel coding. It can be observed that the PESQ-SNR curve of the proposed NST trained under multiple SNRs with SNR token fusion closely approximates the envelope of the curves obtained from models trained using single SNR values.

We additionally carry out experiments on the widely used COST2100 fading channel [23] to verify the robustness of the NST model. Figure 8 shows the results. With a feedback of average SNR and the SNR token fusion, our model adapts to the channel states well, while performances of DeepSC-S are evaluated on models trained at multiple SNRs. With lower bandwidth in Figure 8, NST also shows better transmission efficiency compared to traditional methods.

The subjective user rating results in Figure 9 verify that the proposed NST recovers perceptually satisfying speech over 6 dB AWGN channels, even consuming much less bandwidth than separation-based speech coding methods. Compared to DeepSC-S, which is only optimized for lower distortion, the RDP-optimized NST achieves semantics-guided dynamic rate allocation, thus much improving the end-to-end system gain. The perceptual quality of streaming NST exhibits no substantial degradation compared to the non-streaming one, which is of practical value in RTC scenarios.

3.4. Discussion on the Quality–Latency Tradeoff

In terms of streaming NST, we investigate the tradeoff between the perceptual quality and the transmission latency. As is defined previously, each frame of speech feature ${\mathit{y}}_i$ accounts for 8 milliseconds (ms) of a 16 kHz signal.

Table 1. Quality–delay tradeoff for the streaming NST model tested with SNR = 10 dB over the AWGN channel.

Stride	PESQ	Total Delay	Runtime	Maximum Latency
2	4.09	67.1 ms	51.1 ms	16 ms
3	4.13	83.2 ms	59.2 ms	24 ms
5	4.15	112.7 ms	72.7 ms	40 ms
7	4.17	140.1 ms	84.1 ms	56 ms

shows the tradeoff between speech quality and transmission delay, which consists of encoding and decoding time (runtime) and the latency. In the context of sliding-window-based inference, a longer stride will increase the latency as it needs to wait for the arrival of future frames to collect all features belonging to the same window.

We also compare the PESQ performances versus stride frames across different SNRs and bandwidth cost. The results in Figure 10 verify that a longer window stride as well as the length of contextual windows in JSCC consistently presents a better coding gain across different transmission conditions at the cost of longer delay according to Table 1. Except this subsection, performances of streaming NST are reported with a stride of $N=3$ and a total delay of less than 100 ms. It satisfies the real-time property and ensures a high-quality speech restoration simultaneously. The runtime is evaluated on an Intel(R) Core i9-12900K CPU (Intel Corporation, Santa Clara, CA, USA).

Table 2. Model complexity comparison.

Model	GFLOPs	#Params (Unit: Million)
[3]	>31	>106
DeepSC-S [1]	7.60	0.24
NST (Ours)	9.87	2.49

presents the model complexity comparison of both computational (measured by giga floating point operations per second, i.e., GFLOPs) and space complexity. Due to the employment of a tiny Transformer in the joint source–channel encoder, our model is comparably lightweight and computational efficient. Extra measures for accelerating inference may be taken to facilitate lightweight deployment in resource-limited devices.

4. Conclusions

In this paper, we present the NST, which is a novel neural speech transmission framework. The model features dynamic rate allocation for variable-length JSCC, which is guided by the variational modeling of speech latent features. It presents good adaptability to varying channel conditions by channel information fusing in JSCC. A streaming variant of NST is also designed for RTC. Simulation results verify that the proposed method consumes much less bandwidth cost than classical methods when achieving similar perceptual performances. It highlights NST’s potential in high-efficiency and high-fidelity speech transmission in the realm of semantic communication.