Reinforcement Learning-Based Layered Lossy Image Semantic Coding

Abstract

Semantic communication has garnered increasing attention due to its ability to shift the focus from pixel-level transmission to the transfer of fundamental semantic information representing the core content of images. This paper proposes a novel reinforcement learning-based layered semantic coding (RL-SC) method aimed at optimizing image communication systems under bandwidth constraints. By leveraging deep reinforcement learning (DRL), the proposed method efficiently allocates semantic bits to maximize semantic fidelity while minimizing bitrates. By incorporating semantic segmentation and image reconstruction networks, the framework utilizes both semantic maps and residual information to enhance the image coding process. Experiments demonstrate that the proposed method outperforms traditional image compression techniques and layered image encoding methods without reinforcement learning in preserving semantic content and achieving high-quality image reconstruction. In particular, the proposed method excels in low-bitrate scenarios, effectively maintaining both semantic accuracy and perceptual quality.

1. Introduction

Since Claude Shannon introduced information theory, he and Warren Weaver have continued to develop and refine its underlying principles, ultimately defining three levels of communication: the bit level, the semantic level, and the effective level [1]. Traditional communication paradigms focus on accurate bit transmission while often overlooking the semantic correlations within the transmitted information. As communication technologies advance rapidly and data exchange grows exponentially, conventional systems struggle to meet the requirements of high efficiency and low latency. Moreover, by relying solely on raw data transmission, these systems fail to capture the underlying semantic significance of the conveyed information. In contrast, semantic communication transmits only task-relevant semantic content rather than complete raw data. By adopting a semantic-aware framework, semantic communication systems can significantly reduce data volume while maintaining superior performance in subsequent tasks, including image reconstruction and semantic segmentation.

In bit-level image communication, traditional methods such as JPEG [2] and J2K [3] achieve compression through frequency domain transformation and quantization. More advanced approaches, such as BPG [4] and JPEG XL, further enhance compression efficiency by incorporating advanced entropy coding and perceptual metrics. However, severely compressed images inherently distort human perception and often suffer from degradations, including blocking artifacts, ringing, blurring, and checkerboard artifacts [5,6], which can negatively impact semantic tasks, including segmentation, classification, and detection [7,8]. Consequently, blindly minimizing pixel-level distortion may result in excessive bit overhead.

Recently, image compression methods based on deep learning have been extensively studied and applied. Refs. [9,10,11,12] utilize neural networks, including autoencoders and generative adversarial networks (GANs), to model high-dimensional image distributions, allowing for more efficient and adaptive compression. Toderici et al. [13] pioneered an end-to-end optimized image compression method employing recurrent neural networks (RNN) for image reconstruction. Balle et al. [14] introduced an image compression framework leveraging the variational autoencoder (VAE) architecture. Rippel et al. [15] pioneered the use of adversarial loss functions within an end-to-end framework to enhance the visual quality of compressed images.

The rapid development of artificial intelligence has led to the emergence of semantic communication as a new research frontier in information and communication, which has garnered widespread interest. The core concept of semantic communication is to transmit information based on its semantic content rather than relying solely on bitstreams, aiming to improve communication efficiency and reduce redundant information transmission. Recent studies have investigated the applications of semantic communication in various domains, including natural language processing, image transmission, and multimodal tasks, demonstrating its significant advantages in low-bandwidth and high-latency scenarios.

Firstly, many studies have attempted to integrate deep learning models with semantic communication, leveraging the automatic learning of high-level semantic representations to optimize information transmission. As introduced in [16,17], text-based semantic communication systems enable intelligent exchanges between humans and machines, as well as machine-to-machine interactions. The concept of lossless semantic data compression is explored in [18], proposing that substantial compression can be achieved at the semantic level.

Early attempts at semantic image communication [19,20,21] have developed joint source channel coding (JSCC) methods and achieved promising performance under challenging conditions such as low signal-to-noise ratio (SNR) and constrained bandwidth. For instance, Ref. [19] directly mapped image pixel values to complex-valued channel input symbols and learned noise-robust encoded representations, outperforming traditional separate digital communication methods across all SNRs. Subsequently, Kurka et al. [20] introduced a multi-description JSCC scheme designed for bandwidth-adaptive image transmission. Attention-based JSCC image transmission [21] effectively enabled robust operation across varying SNRs during transmission. Employing semantic labels, DSSLIC [22] proposed a framework for deep-layered image compression. However, this method primarily focuses on traditional pixel-level fidelity, which limits its capability to effectively reconstruct semantic information.

In the design of efficient communication systems, resource allocation optimization is critical, and reinforcement learning (RL) has played a significant role in this area. In particular, deep Q networks (DQN) [23] have emerged as powerful tools for addressing sequential decision-making problems, as they can approximate complex value functions using deep neural networks. DQN is especially well suited for tasks that require adaptive optimization, such as semantic bit allocation in communication systems, where it dynamically adjusts parameters under changing constraints to maximize system performance.

Task-driven semantic encoding, which adopts a conventional hybrid coding framework, incorporates semantic fidelity metrics into the optimization process and leverages RL to achieve semantic bit allocation. In [24], researchers utilized deep Q-learning to optimize the selection of quantization parameters for various coding units, yielding compelling results for decoded images in tasks such as classification, detection, and segmentation. Ref. [25] introduces RL-ASC, a deep learning-driven image semantic encoding approach that employs RL for adaptive semantic bit allocation and integrates GANs for semantic decoding. This technique effectively reconstructs visually compelling and semantically consistent images, even at low bitrates.

Despite significant progress in semantic communication and image compression, existing hierarchical image encoding methods, such as deep semantic segmentation-based learned image compression (DSSLIC) [22], still face notable limitations. DSSLIC relies on a predefined layered semantic structure to guide bit allocation, yet it lacks adaptive decision-making during the encoding process. The bit-allocation strategy is often manually designed or heuristic-based, making it less flexible when dealing with images of varying semantic importance. As a result, it may over-allocate bits to less critical regions while failing to preserve essential semantic details, leading to suboptimal performance in both rate-distortion efficiency and downstream semantic analysis tasks.

To overcome these challenges, this paper proposes a reinforcement learning-based layered semantic coding (RL-SC) method, which dynamically adjusts quantization levels based on the semantic significance of different image regions. The core idea is to model the bit-allocation process as a Markov Decision Process (MDP), where an agent learns an optimal policy to maintain semantic integrity while improving bitrate effectiveness. Specifically, we employ a DQN to guide the adaptive selection of quantization levels, ensuring that critical semantic components receive higher priority while redundant information is compressed more aggressively. Unlike DSSLIC, which follows a fixed layered structure, the proposed method optimizes bit allocation through an adaptive decision-making process, enabling more flexible adjustments to varying the image content.

The proposed method’s effectiveness is validated by experimental findings, highlighting its capability to optimize semantic image compression by adaptively allocating bits based on semantic importance. The approach effectively balances compression efficiency and semantic fidelity, making it a promising solution for future semantic communication systems.

The remainder of the paper is structured as follows. Section 2 presents the architecture of the proposed RL-SC method. Section 3 describes the proposed RL-based semantic quantization along with its training algorithm. Section 4 introduces the Adaptive Residual Compression method. Section 5 presents the details of the experiment and the performance of the proposed method. Section 6 summarizes the main contributions of this paper and discusses potential future research directions.

2. Reinforcement Learning-Based Layered Image Semantic Coding

The proposed RL-SC is illustrated in Figure 1. At the sender, the semantic map and residual information of the image are first extracted using the semantic segmentation network and the image generation network. These are then input into the RL-based semantic quantizer and the Adaptive Residual Compression module, with the former performing adaptive semantic quantization and the latter generating the final residual. At the receiver, the extracted semantic map is used to generate a synthesized image via a generative adversarial network. This image is subsequently combined with the decoded residual to obtain the final target image. The goal of the system is to maximize semantic fidelity under low-bitrate conditions. Details of the semantic encoder and semantic decoder are provided below.

2.1. Semantic Encoder

At the transmitter, the semantic map of the image is initially extracted using a semantic segmentation network. This network processes the input image and assigns a semantic label to each pixel. Next, we obtain the residual information using an image generator. This network reconstructs the image based on the extracted semantic map, and the residual information is calculated as the discrepancy between the original and the generated image.

Denote the input image as $\mathbf{x}\in R^{3\times H\times W}$, where H and W represent the image’s height and width, respectively, and “3” indicates the number of channels. The input image is processed by the semantic segmentation network to generate the semantic map $\mathbf{s}\in R^{H\times W}$. The value at the coordinate ($x,y$) in $\mathbf{s}$ is denoted as $\mathbf{s}(x,y)$, and $\mathbf{s}(x,y)\in \{1,2,\dots ,N\}$ represents the class label. Next, we utilize an image generator, such as a GAN or other image generation networks, to produce a coarse reconstruction of the image $\mathbf{x}’$ and then compute the residual $\mathbf{r}$ by subtracting $\mathbf{x}’$ from the input image $\mathbf{x}$.

Existing algorithms directly compress and transmit residuals, but this approach fails to fully consider the semantic importance of different regions, leading to inefficient bit allocation. To address this issue, we introduce the RL-based semantic quantizer and residual processing model. The RL-based semantic quantizer leverages reinforcement learning to adaptively adjust the quantization strategy, ensuring that key semantic regions receive finer encoding while less critical information is compressed more efficiently. Meanwhile, the Adaptive Residual Compression module optimizes the encoding of residual information based on semantic relevance, reducing redundancy and enhancing both semantic fidelity and visual quality under low-bitrate conditions. This design enables more efficient bit allocation at the same bitrate, ensuring the effectiveness and robustness of the semantic communication system.

The importance of different semantic concepts varies in downstream analysis tasks. Therefore, our goal is to achieve precise encoding without losing semantic information. Taking the Cityscapes dataset as an example, semantic segmentation and object detection tasks mainly focus on vehicles, pedestrians, and traffic signs, whereas background elements such as the sky and buildings have less impact on the key semantic content of the image. Although traditional compression methods, such as JPEG and BPG, perform well with respect to pixel-based metrics like Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), they often lead to significant distortion of semantic information in the reconstructed image.

To focus on semantic concepts and achieve precise encoding, we propose a semantic encoding method based on deep reinforcement learning. We model the encoding process as an MDP, where the semantic concepts are encoded in the order of labels $i=1,\dots ,N$, with N denoting the total number of semantic classes, by selecting the optimal quantization levels. This approach ensures an aggressive bitrate and an optimal encoding scheme. The RL agent adaptively allocates quantization levels by balancing semantic performance and encoding cost. This process will be explained in detail in Section 3.

To further compress the residual information, we utilize a residual processing model, which mainly consists of two modules: Semantic-based Patch Selection and Selective Zeroing. This method effectively preserves the residual information of the low-proportion categories, thereby improving the semantic performance of the output image while maintaining an aggressive compression ratio. This process will be explained in detail in Section 4.

After determining the optimal quantization levels through an RL-based semantic quantizer, the final step is to encode the quantized semantic concepts into a compressed bit stream. Specifically, the semantic map is losslessly compressed using Huffman coding, ensuring efficient storage and transmission of semantic information. Meanwhile, we adopt the JPEG XL compression framework to encode each semantic concept separately based on its assigned quantization level. This approach ensures that semantically important regions are preserved with high precision, while less critical areas undergo more aggressive compression, effectively balancing bitrate and semantic fidelity.

2.2. Generative Semantic Decoder

At the receiver, the received bitstream is first decoded to reconstruct the semantic map $\mathbf{s}$ and the quantized semantic concepts ${\mathbf{r}}^{\prime }$. Specifically, the semantic map $\mathbf{s}$ is losslessly decoded using Huffman coding, ensuring that the transmitted semantic segmentation information is accurately recovered. Meanwhile, the residual ${\mathbf{r}}^{\prime }$ is decoded using the JPEG XL framework, which efficiently restores the compressed residual information while maintaining a balance between compression efficiency and reconstruction quality.

Once the semantic map and quantized semantic concepts are reconstructed, an image synthesis network is used to generate the reconstructed image $\mathbf{x}’$. Unlike traditional decoding methods, which primarily focus on pixel-wise reconstruction, semantic communication emphasizes the preservation of meaningful content. To achieve this, the image synthesis network leverages the semantic map $\mathbf{s}$ to guide the generation of a visually and semantically coherent image.

A variety of generative models can be used for this purpose, including GAN-based architectures, which have demonstrated strong capabilities in generating high-resolution images with complex structures. Specifically, GANs aim to capture the underlying distribution of real-world images, making them well suited for semantic-guided image reconstruction. For instance, LGGAN [26] is one possible approach that integrates both local class-specific and global image-level features to enhance scene generation. However, our framework is not limited to a specific generative model, providing flexibility in selecting different architectures based on performance requirements and computational constraints.

In this paper, we employ a pre-trained image generation network, where the semantic map $\mathbf{s}$ serves as input, conditioning the synthesis of the reconstructed image ${\mathbf{x}}^{\prime }$. The decoded quantized semantic concepts ${\mathbf{r}}^{\prime }$ are then added to the output of the image generator to produce the final reconstructed image.

3. Reinforcement Learning-Based Semantic Quantization

This section presents our proposed RL-based semantic quantization model, designed to optimize semantic encoding and residual compression for image communication systems. The model adaptively allocates bitrates to different categories based on their importance in the image, using RL to optimize the encoding process by balancing compression efficiency and semantic fidelity. The architecture and training procedure of the model are detailed in the following sections.

3.1. Architecture of the RL-Based Semantic Quantization Model

Traditional image communication methods primarily evaluate images at the pixel level, requiring the reconstructed image to be identical to the original at the pixel level after transmission. However, semantic communication shifts the focus from pixel-wise accuracy to semantic fidelity and coding efficiency, allowing for certain pixel-level deviations as long as the core semantic information is preserved.

A common approach to encoding semantic information involves heuristic methods, such as threshold-based schemes, which allocate higher bitrates to critical semantic components. However, these methods rely heavily on manual design and lack adaptability, making them less efficient for complex image communication tasks. To overcome these limitations, we propose an RL-based semantic quantization model that adaptively allocates bitrates based on the importance of semantic concepts. Specifically, we employ DQN to optimize the semantic quantization process. The architecture of the proposed RL-based semantic quantization model is illustrated in Figure 2.

We define ${\mathbf{s}}_i$ as the category semantic mask, where $i=1,2,\dots ,N$, and its entries are represented as follows:

$${\mathbf{s}}_i(x,y)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathbf{s}(x,y)=i,\hfill \\ 0,\hfill & otherwise.\hfill \end{array}\right.$$

(1)

The residual $\mathbf{r}$ is element-wise multiplied with the category semantic mask to obtain the category residual ${\mathbf{r}}_i$:

$${\mathbf{r}}_i=\mathbf{r}\odot {\mathbf{s}}_i$$

(2)

where ⊙ denotes element-wise multiplication.

To obtain an optimal encoding scheme, the encoding process is modeled as an MDP. In this process, the agent observes the current state and selects an appropriate quantization level for compressing and encoding the category residual ${\mathbf{r}}_i$, executing the process sequentially according to the label index from $i=1$ to $i=N$. This process includes five key elements: state, action, reward, state transition probability, and policy. The agent’s behavior is defined as a policy. Since the state transitions are deterministic, the state transition probability is always 1. The details are as follows:

(1) State: To decide on the next action, the agent must collect and analyze all relevant information from the current environment. At step i, the state is defined as $stat{e}_i=[{\mathbf{r}}_i,{\mathbf{s}}_i,{\mathbf{I}}_i]$, which consists of the category semantic mask, category residual, and category information. Here, ${\mathbf{I}}_i$ is a $N\times 1$ one-hot vector representing the category information, where the i-th element is set to 1.

(2) Action: The action space A is defined as a set of quantization levels, $A=\{q_1,q_2,\dots ,q_m\}$, where m refers to the total number of levels. In traditional image coding methods such as JPEG/JPEG XL, the quantization level directly determines the number of coding bits and coding quality. Higher quantization levels correspond to higher bitrates and better reconstruction quality. At each state $stat{e}_i$, the agent selects an action $a_i$ from the action space A, determining the quantization level for the corresponding category residual. This decision is made based on the agent’s observations, enabling adaptive encoding tailored to different semantic concepts. Once $a_i$ is determined, ${\mathbf{r}}_i$ is compressed using methods such as JPEG XL.

(3) Reward: The cumulative reward, as a measure of action evaluation, acts as the optimization objective in MDP. By observing the rewards, the agent is directed to learn the optimal behavior. In the semantic communication system, our goal is to retain as much semantic information as possible while maintaining a low bitrate. At the i-th step, the semantic loss of the reconstructed image can be expressed as:

$$L_i=\lambda L_r+L_s+\gamma L_p$$

(3)

where $L_r$ denotes the bits per pixel (BPP) of the encoded class mask for the i-th category, which is the ratio of the encoded bitstream length to the total number of pixels in the image. $L_s$ represents the semantic loss, which measures the difference between the reconstructed image’s labels and the ground truth labels, while $L_p$ indicates the perceptual loss that reflects the perceptual quality of the reconstructed image. These losses will be formally defined in the next subsection. $\lambda$ and $\gamma$ are hyper parameters. The agent can learn different encoding strategies by adjusting them, thereby optimizing various semantic encoding models.

In the MDP process, our reward is defined as the reduction in semantic loss from $stat{e}_i$ to $stat{e}_{i+1}$, which can be expressed as:

$$R_i=L_i-L_{i+1}$$

(4)

The agent’s goal is to maximize the cumulative reward G throughout the episode, where $\delta$ represents the discount factor, and the discounted cumulated reward can be expressed as:

$$G=\sum _{i=1}^N{\delta }^i{R}_{i+1}$$

(5)

(4) Agent: We design an agent capable of observing the environmental state and executing a sequence of actions for semantic encoding. To accomplish this, we adopt the DQN algorithm, which enables the agent to learn an optimal quantization policy through trial and error. In this paper, the agent predicts the optimal action through the Q-network. The architecture of the Q-network is illustrated in Figure 2. Initially, ${\mathbf{r}}_i$ from the state is processed through two consecutive convolutional layers, each followed by a max-pooling operation. The output of the convolutional layers is subsequently flattened into a one-dimensional vector and concatenated with ${\mathbf{I}}_i$. The resulting vector is passed through three fully connected layers, ultimately producing an $m\times 1$ vector. All convolutional layers are activated using Leaky Rectified Linear Units (Leaky ReLU), while the output layer utilizes a SoftMax function to compute the selection probabilities for each action. To reduce overfitting, a dropout layer is added to the network.

By inputting the $stat{e}_i$, the Q-network outputs the discounted cumulative reward for each action a as $Q(stat{e}_i,a)$. The optimal action can then be obtained using the following formula:

$$a_i=\arg \underset{a}{\max }Q(stat{e}_i,a)$$

(6)

3.2. Semantic Perceptual Loss

Traditional image communication systems commonly use PSNR and SSIM to evaluate image reconstruction quality. However, semantic communication typically focuses on transmitting the underlying semantic information and features of the image. In the previous section, we introduced the semantic loss function in (3), which balances the trade-off among bitrate $L_r$, semantic fidelity $L_s$, and perceptual quality $L_p$.

The semantic loss $L_s$ is calculated using the Mean Intersection over Union (MIoU), which quantifies the discrepancy between the reconstructed image’s labels and the ground truth labels. Formally, the semantic loss $L_s$ is expressed as:

$$L_s=1-MIoU$$

(7)

MIoU is a commonly employed metric in semantic segmentation, measuring the average classification accuracy across all semantic categories. It is computed as:

$$MIoU={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}Io{U}_i$$

(8)

where N represents the total number of semantic categories, and ${\mathit{IoU}}_i$ denotes the Intersection over Union (IoU) for class i. The IoU for each class i is given by:

$${IoU}_i={\displaystyle \frac{T{P}_i}{T{P}_i+F{P}_i+F{N}_i}}$$

(9)

where the metrics $T{P}_i$, $F{P}_i$, and $F{N}_i$ quantify the model’s performance for class i. Specifically, $T{P}_i$ denotes the number of pixels correctly classified as belonging to class i, while $F{P}_i$ denotes the pixels mistakenly assigned to that class. Conversely, $F{N}_i$ corresponds to the pixels that should have been classified as class i but were misidentified.

Minimizing $L_s$ naturally maximizes MIoU, ensuring that the reconstructed image retains its semantic fidelity. The MIoU computes the Intersection over Union (IoU) for each class and derives an overall score by averaging these values across all classes. This metric thoroughly evaluates how accurately each pixel is classified, taking into account correct classifications as well as misclassifications. Consequently, MIoU provides a comprehensive assessment of the model’s semantic reconstruction performance across N classes. In this work, we use a pre-trained [27] to obtain semantic segmentation predictions for calculating MIoU.

Although MIoU effectively quantifies the reconstruction quality at the semantic level and performs well in semantic communication, it primarily focuses on the recovery of each semantic class and lacks a comprehensive capture of the high-level structural information and perceptual effects of the image. To address this limitation, we introduce perceptual loss $L_p$. Pre-trained VGG [28] networks can be used to extract high-dimensional features of the image, and these features are then used to evaluate the perceptual similarity of the image. The Euclidean distance between the feature maps of the original and the reconstructed images at these layers is calculated as the perceptual loss. Specifically, the perceptual loss is computed using the following formula:

$$L_p={∥\phi \left(\mathbf{x}\right)-\phi \left({\mathbf{x}}^{\prime }\right)∥}^2$$

(10)

here, $\phi \left(\mathbf{x}\right)$ and $\phi \left({\mathbf{x}}^{\prime }\right)$ represent the feature maps of the original image $\mathbf{x}$ and the reconstructed image ${\mathbf{x}}^{\prime }$. In this way, perceptual loss can effectively capture the semantic information and high-level structural details of the image, thereby improving image reconstruction quality and making the reconstructed image visually closer to the perceptual effect of the real image.

3.3. Training Algorithm

During training, we adopt the DQN [23] training framework and approximate the Q-value function using a deep neural network. We initialize the Q-network and the target Q-network and use an experience replay buffer to store the agent’s interaction experiences with the environment.

In the initialization step of each training round, we input the semantic map into the image generator to obtain the initial reconstructed image ${\mathbf{x}}_0$. By feeding $stat{e}_i$ into the agent, we can adaptively obtain the corresponding quantization parameters. We denote ${\mathbf{x}}_{i-1}$ as the reconstructed image from the previous step. The semantic decoder processes the received bitstream to obtain the category residual ${\mathbf{r}}_{i-1}$, which is then added to ${\mathbf{x}}_{i-1}$ to obtain the current reconstructed image ${\mathbf{x}}_i$. The image ${\mathbf{x}}_i$ is fed into both the semantic segmentation network and the VGG network to obtain the semantic loss $L_i$ for the current state. Based on (4), the reward $R_i$ is obtained, indicating whether the selected action is beneficial. The reward is determined by the reduction in perceptual loss, semantic loss, and bitrate after executing the action.

At each training step, the agent follows the $ϵ$-greedy policy to select an action, execute it, and then store the transition $(stat{e}_i,a_i,R_i,stat{e}_{i+1})$ in the replay buffer. Then, a batch of experiences is randomly sampled to calculate the target Q-values. The network parameters are updated by minimizing the error between the Q-network output and the target values. Meanwhile, the target Q-network parameters are periodically synchronized to ensure training stability. This iterative process approximates the optimal policy, enabling efficient decision-making. The training process of the RL-based semantic quantization model is outlined in Algorithm 1.

Algorithm 1 Training Process

Input: Semantic map $\mathbf{s}$, residual $\mathbf{r}$, dataset X, action space A, training epoch $E_p$, discount

factor $\delta$, update interval C, total number of semantic classes N

Output: Q-network parameter $\theta$   1: Initialize the Q-network parameters $\theta$ and target Q-network parameters ${\theta }^{\prime }$   2: for $epoch=1$ to $E_p$ do   3:     for x in X do   4:         for $i=1$ to N do   5:             Set state $stat{e}_i$   6:             Choose action $a_i=\arg {\max }_{a}Q(stat{e}_i,a)$   7:             Encode $r_i$ with quantization levels $q=A\left[a_i\right]$   8:             Obtain next state ${state}_{i+1}$   9:             Compute semantic loss $L_i$ and $L_{i+1}$ by (3) 10:             Compute reward $R_i$ by (4) 11:             Store transition $(stat{e}_i,a_i,R_i,stat{e}_{i+1})$ 12:             Sample a mini-batch of transition from buffer $(stat{e}_{i^{\prime }},a_{i^{\prime }},R_{i^{\prime }},stat{e}_{i^{\prime }+1})$ 13:             Compute $y_{i^{\prime }}=R_{i^{\prime }}+\delta ma{x}_{a}Q(stat{e}_{i^{\prime }+1},a;\theta )$ 14:             Update $\theta$ with the loss ${\sum }_{i^{\prime }}{[-Q(stat{e}_{i^{\prime }},a_{i^{\prime }};\theta )]}^2$ 15:             Every C steps, synchronize ${\theta }^{\prime }$ = $\theta$ 16:         end for 17:     end for 18: end for 19: Return the Q-network parameter $\theta$.

4. Residual Processing Module

To further enhance the compression efficiency of residual information while preserving the semantic representation of the image, we adopt a residual processing method proposed in our previous work [29]. This method leverages semantic information to perform differentiated processing based on the pixel proportions of different categories, with an emphasis on preserving the residuals of categories that have a greater impact on semantic recognition but occupy a smaller proportion in the image. As a consequence, it enhances the semantic fidelity of the reconstructed image while maintaining a reasonable compression ratio. As presented in Figure 3, the module consists of two parts: Semantic-based Patch Selection and Selective Zeroing.

Semantic-based Patch Selection: Different semantic categories in an image exhibit significant spatial imbalance. For example, in the Cityscapes dataset, the number of pixels belonging to the road category is significantly larger than that of the streetlight category. Consequently, for high-proportion categories such as road, slight deviations in pixel values may not significantly impact the recognition performance of the semantic segmentation network. However, for low-proportion categories such as streetlight, deviations in pixel values can severely affect semantic recognition. Therefore, residual processing should prioritize the retention of residuals for categories that have a greater impact on semantic performance while suppressing residuals for high-proportion categories to enhance compression efficiency.

First, we calculate the proportion of each semantic category in the image and classify categories with proportions below a predefined threshold as low-proportion categories. Then, through the patch selection process, a binary matrix $M_{high}$ with the same size as the original image is generated, which is divided into $32\times 16$ patches. Patches containing low-proportion categories are assigned a value of 0, while the remaining patches are set to 1, effectively distinguishing between high-proportion and low-proportion regions.

Selective Zeroing: To further enhance residual compression efficiency, we introduce the selective zeroing strategy. This approach refines the residual representation by eliminating insignificant values, thereby reducing redundancy while preserving critical information. The fundamental principle is to set residuals with absolute values below a predefined threshold to zero, increasing the proportion of zero values in the overall residual map. The residual $\mathbf{r}$ is element-wise multiplied by the binary mask $M_{high}$ to obtain ${\mathbf{r}}_{high}$, which contains only the residuals of high-proportion categories. Then, elements in ${\mathbf{r}}_{high}$ with an absolute value smaller than a predefined threshold are set to zero, effectively removing insignificant residual information. This strategy ensures a high compression ratio, even under lossy compression with high-quality factors, while mitigating the adverse effects of lossy encoding and decoding on low-proportion categories.

By combining these two methods, we are able to balance compression efficiency and semantic reconstruction performance, ensuring that the semantic information in the image is maximized during the reconstruction process.

5. Experiments

This section provides a detailed description of the dataset, training settings, and the performance of the proposed image semantic coding method. It also discusses the effect of varying compression rates on the semantic transmission of images.

5.1. Experimental Settings

Datasets: The experiments in this paper were conducted on the Cityscapes dataset, a widely used benchmark for urban street scene image semantic segmentation tasks. The dataset contains up to 5000 high-quality pixel-level annotated images, containing 2975 training images, 500 validation images, and 1525 test images. It includes 30 categories, primarily covering various objects and scenes in urban environments such as buildings, roads, pedestrians, and vehicles, making it ideal for assessing the performance of semantic segmentation and image reconstruction models. We downsampled the images and semantic labels to a size of 256 × 512 and conducted testing on the validation set.

Training Settings: In the simulations, the DQN training parameters are randomly initialized. The learning rate is set $\alpha =0.0001$, and the discount factor is $\delta =0.9$. The $ϵ$-greedy strategy is employed for action selection, where $ϵ$ is initialized to 0.9. The experience replay buffer has a capacity of 10,000, with a mini-batch size of 1. The target network parameters are updated every 50 steps. Training is conducted over $E_p=10$ episodes. The action space is defined as $A=\{0,60,80,90,99\}$. The total number of semantic classes is set to $N=19$. The model is implemented using the PyTorch framework (version 1.10.1).

Baselines: We compare the proposed method with standard image compression techniques such as JPEG, J2K, JPEG XL, and BPG, as well as with the deep learning-based image codec DSSLIC [22] and MLIC [30].

Evaluation Metrics: To comprehensively assess the model’s performance, MIoU was used as an objective metric, where a higher MIoU value, up to 1, indicates less semantic information loss. To ensure fair comparisons across different codecs, including traditional methods, learning-based models, and the proposed RL-SC method, a pre-trained semantic segmentation model was uniformly applied to the reconstructed images to generate predicted segmentation maps. These were then compared with ground truth labels to compute MIoU, enabling a consistent evaluation of semantic fidelity. In addition, kernel inception distance (KID) [31] and Fréchet inception distance (FID) [32], which align closely with human perception, were employed to measure the similarity between the reconstructed and input images in the deep feature space. Specifically, KID and FID measure the distributional differences between reconstructed images and real samples through the Inception network. These metrics are commonly used in the context of GANs to measure the quality and diversity of generated samples.

5.2. Results and Analysis

In Figure 4, we compared various image compression methods at different BPP using MIoU as the primary evaluation metric. The results confirm that the proposed method outperforms all alternatives across the entire bitrate range.

Specifically, at an extremely low bitrate (0.14 BPP), the proposed RL-SC achieves an MIoU of approximately 0.46, which is significantly higher than that of other methods. This demonstrates that the proposed RL-SC can more effectively preserve semantic information under extremely low-bitrate conditions. Meanwhile, at a high bitrate (0.4 BPP), the proposed RL-SC achieves an MIoU close to 0.6, significantly outperforming DSSLIC (0.3) and BPG (0.38), exhibiting stronger semantic awareness compared to traditional compression methods.

These results indicate that the proposed RL-SC effectively reduces the loss of semantic information at low bitrates and further enhances semantic reconstruction performance at high bitrates. This can be attributed to the introduction of RL-based semantic quantization, which allocates encoding resources to different semantic categories, achieving an effective balance between semantic preservation and compression efficiency.

In Figure 5 and Figure 6, we compared the perceptual quality of various image compression methods at different BPP using KID and FID as evaluation metrics. Lower KID and FID values indicate smaller distribution discrepancies between the reconstructed and original images, reflecting higher perceptual quality.

The experimental results demonstrate that the proposed RL-SC outperforms other image compression methods across all bitrate ranges. Specifically, at a low bitrate (0.14 BPP), the proposed RL-SC achieved a KID of approximately 0.03 and an FID of around 70, whereas BPG reported a KID of approximately 0.04 and an FID close to 100. Traditional methods, such as J2K, exhibited a KID exceeding 0.08 and an FID surpassing 150. These findings indicate that, under extremely low-bitrate conditions, the proposed RL-SC is significantly more effective in preserving the perceptual quality of images. Furthermore, at a higher bitrate (0.4 BPP), the proposed RL-SC achieved a KID of approximately 0.01 and an FID of around 60, indicating strong perceptual performance. Although MLIC slightly outperforms RL-SC in terms of KID when BPP exceeds 0.25, RL-SC consistently demonstrates competitive results and superior perceptual fidelity compared to traditional codecs and other learning-based methods across the entire bitrate range.

This performance improvement can be attributed to the introduction of perceptual loss during the encoding process, enabling the model to effectively capture semantic information and high-level structural details of the image. Consequently, the quality of reconstructed images is significantly enhanced, with perceptual effects closely resembling those of the original images. Furthermore, the proposed RL-SC integrates a GAN architecture and employs adversarial loss, which contributes to the naturalness and realism of reconstructed images, clearly validating the effectiveness of the designed RL-based coding strategy.

To evaluate the rate-distortion performance of the proposed RL-SC framework, we report the PSNR and SSIM across various bitrates, as shown in Figure 7. While these metrics are standard in traditional image coding systems, they primarily emphasize pixel-wise fidelity and structural similarity. As expected, baseline methods achieve higher PSNR and SSIM values, since they are directly optimized for these distortion metrics. In contrast, RL-SC is designed to preserve semantic content under bandwidth constraints and is inherently tolerant to pixel-level distortions. Unlike conventional methods that enforce local consistency, our approach focuses on maintaining global semantic integrity, which inevitably leads to a moderate degradation in pixel-level metrics. Nonetheless, this trade-off is aligned with the objective of semantic communication, where the preservation of meaningful content is prioritized over precise pixel reconstruction.

This study further evaluates the memory overhead and scalability of the model during the inference stage. The experiments were conducted on an NVIDIA RTX 1080 GPU and an Intel Core i9-7900X CPU (NVIDIA Corporation, Santa Clara, CA, USA; Intel Corporation, Santa Clara, CA, USA). The proposed encoding framework has a peak memory consumption of approximately 2.5 GB when compressing a single image. These results demonstrate that the proposed framework achieves a reasonable trade-off between performance and computational cost, making it suitable for offline image compression tasks under moderate hardware constraints.

The results above validate that the proposed RL-SC not only preserves semantic consistency but also significantly enhances perceptual quality, especially at low bitrates, demonstrating superior performance and practical value.

5.3. Ablation Study

To evaluate the effectiveness of individual components in the proposed semantic compression framework, we conducted two ablation experiments: one to assess the contribution of the RL-based quantization and the other to evaluate the impact of the residual processing module.

To examine the effectiveness of the reinforcement learning (RL)-based quantizer, a comparative experiment was conducted by replacing it with a fixed quantization strategy. In the fixed setting, a constant quantization level was applied uniformly across all residual components, without considering semantic importance. As illustrated in Figure 8, the proposed RL-based quantizer consistently achieves higher mean Intersection over Union (MIoU) values across different bit-per-pixel (BPP) settings compared to the fixed approach. This improvement is particularly noticeable under lower BPP conditions, where efficient bit allocation becomes crucial. The adaptive quantization policy learned via RL focuses bits on semantically significant regions, leading to a more accurate reconstruction in terms of semantic fidelity. These findings underscore the contribution of the RL module to the overall performance of the framework.

The second ablation experiment aimed to evaluate the influence of the residual processing module on semantic reconstruction performance. This study compares the proposed method, which integrates the residual processing module, with an ablated variant that excludes it entirely. By removing residual processing, the system no longer benefits from selective zeroing or semantic-guided filtering of less significant information. The experimental results, presented in Figure 9, show a noticeable decline in performance when the residual processing module is excluded. This finding confirms that the residual processing module is instrumental in balancing compression efficiency with semantic fidelity. By discarding uninformative residuals while retaining significant details, the module helps maintain higher accuracy in downstream semantic tasks, thereby reinforcing its value within the overall framework.

6. Conclusions

In this paper, we propose an RL-based layered image semantic coding method for semantic communication. By integrating semantic communication, advanced image compression, and reinforcement learning, RL-SC adaptively allocates semantic bits using a DQN-based strategy, effectively minimizing semantic distortion under bandwidth constraints. Perceptual and semantic loss further enhance reconstruction quality, ensuring the preservation of both low-level details and high-level semantic content. Extensive experiments on the Cityscapes dataset validate the superiority of RL-SC. Evaluated using MIoU, FID, and KID, it outperforms traditional compression methods and learning-based codecs such as DSSLIC and MLIC. The results demonstrate that reinforcement learning enables more efficient semantic encoding by dynamically optimizing bit allocation, preserving essential semantic information while maintaining efficiency under limited bandwidth.