Semantic Information Recovery in Wireless Networks

Abstract

Motivated by the recent success of Machine Learning (ML) tools in wireless communications, the idea of semantic communication by Weaver from 1949 has gained attention. It breaks with Shannon’s classic design paradigm by aiming to transmit the meaning of a message, i.e., semantics, rather than its exact version and, thus, enables savings in information rate. In this work, we extend the fundamental approach from Basu et al. for modeling semantics to the complete communications Markov chain. Thus, we model semantics by means of hidden random variables and define the semantic communication task as the data-reduced and reliable transmission of messages over a communication channel such that semantics is best preserved. We consider this task as an end-to-end Information Bottleneck problem, enabling compression while preserving relevant information. As a solution approach, we propose the ML-based semantic communication system SINFONY and use it for a distributed multipoint scenario; SINFONY communicates the meaning behind multiple messages that are observed at different senders to a single receiver for semantic recovery. We analyze SINFONY by processing images as message examples. Numerical results reveal a tremendous rate-normalized SNR shift up to 20 dB compared to classically designed communication systems.

1. Introduction

When Shannon laid the theoretical foundation of the research area of communications engineering back in 1948, he deliberately excluded semantic aspects from the system design [1,2]. In fact, the idea of addressing semantics in communications arose shortly after Shannon’s work in [2], but it remained largely unexplored. Since then, the design focus of communication systems has been on digital error-free point-to-point symbol transmission.

Today, the systems already operate close to the Shannon limit calling for a paradigm shift towards including semantic content of messages in the system design. For example, the data traffic growth still continues with the emergence of the Internet-of-Everything including, e.g., autonomous driving and virtual reality, and cannot be managed by semantics-agnostic communication as it limits the achievable efficiency in terms of bandwidth, power, latency, and complexity trade-offs [3]. Other notable examples include wireless sensor networks, broadcast scenarios, and non-ergodic channels where separation of source and channel coding according to Shannon’s digital design paradigm is generally suboptimal [4,5].

Owing to the great success of Artificial Intelligence (AI) and, in particular, its subdomain Machine Learning (ML), ML tools have been recently investigated for wireless communications and has shown promising application for improving the performance complexity trade-off [6,7,8]. Now, ML with its ability to extract features appears to be a proper means to realize a semantic design. Further, we note that the latter design is supported and possibly enabled by the 6G vision of integrating AI and ML on all layers of the communications system design, i.e., by an ML-native air interface.

Motivated by these new ML tools, and driven by the unprecedented needs of the next wireless communication standard, 6G, in terms of data rate, latency, and power, the idea of semantic communication has received considerable attention [2,9,10,11,12,13]. It breaks with the existing classic design paradigms by including semantics in the design of the wireless transmission. The goal of such a transmission is, therefore, to deliver the required data from which the highest levels of quality of information may be derived, as perceived by the application and/or the user. More precisely, semantic communication aims to transmit the meaning of a message rather than its exact version and hence enables compression and coding to the actual semantic content. Thus, savings in bandwidth, power, and complexity are expected.

In the following, we first summarize in Section 2 related work on semantic communication and justify our main contributions in Section 3. In Section 4.1, we reinterpret Weaver’s philosophical considerations paving the way for our proposed theoretical framework in Section 4. Finally, in Section 5 and Section 6, we provide one numerical example of semantic communication, i.e., SINFONY, and summarize the main results, respectively.

2. Related Work

The notion of semantic communication traces back to Weaver [2] who reviewed Shannon’s information theory [1] in 1949 and amended considerations with regard to semantic content of messages. Often quoted is his statement that “there seem to be [communication] problems at three levels” [2]:

A.

How accurately can the symbols of communication be transmitted? (The technical problem.)

B.

How precisely do the transmitted symbols convey the desired meaning? (The semantic problem.)

C.

How effectively does the received meaning affect conduct in the desired way? (The effectiveness problem.)

Since then semantic communication was mainly investigated from a philosophical point of view, see, e.g., [14,15].

The generic model of Weaver was revisited by Bao, Basu et al. in [16,17] where the authors define semantic information source and semantic channel. In particular, the authors consider a semantic source that “observes the world and generates meaningful messages characterizing these observations” [17]. The source is equivalent to conclusions, i.e., “models” of the world, that are unequivocally drawn following a set of known inference rules based on observation of messages. In [16], the authors consider joint semantic compression and channel coding at Level B with the classic transmission system, i.e., Level A, as the (semantic) channel. In contrast, [17] only deals with semantic compression and uses a different definition of the semantic channel (which we will make use of in this article): It is equal to the entailment relations between “models” and “messages”. By this means, the authors are able to derive semantic counterparts of the source and channel coding theorems. However, as the authors admit, these theorems do not tell how to develop optimal coding algorithms and the assumption of a logic-based model-theoretical description leads to “many non-trivial simplifications” [16].

In [18], the authors follow a different approach in the context of Natural Language Processing (NLP). They define semantic similarity as a semantic error measure using taxonomies, i.e., human knowledge graphs, to quantify the distance between the meanings of two words. Based on this metric, communication of a finite set of words is modeled as a Bayesian game from game theory and optimized for improved semantic transmission over a binary symmetric channel.

Recently, drawing inspiration from Weaver, Bao, Basu et al. [2,16,17] and enabled by the rise of ML in communications research, Deep Neural Network (DNN)-based NLP techniques, i.e., transformer networks, were introduced in AutoEncoders (AEs) for the task of text transmission [19,20,21]. The aim of these techniques is to learn compressed hidden representations of the semantic content of sentences to improve communication efficiency, but the exact recovery of the source (text) is the main objective. The approach improves performance in semantic metrics, especially at low SNR compared to classical digital transmissions. It has been adapted to numerous other problems, e.g., speech transmission [22,23] and multi-user transmission with multi-modal data [24]. Even knowledge graphs, i.e., a prior knowledge base, were incorporated into the transformer-based AE design to improve inference at the receiver side and, thus, text recovery [25].

Not considering Weaver’s idea of semantic communication in particular, the authors in [26] show, for the first time, that task-oriented communications (Level C) for edge cloud transmission can be mathematically formulated as an Information Bottleneck (IB) optimization problem. Moreover, for solving the IB problem, they introduce a DNN-based approximation and show its applicability for the specific task of edge cloud transmission. The terminus “semantic information” is only mentioned once in [26] referring to Joint Source-Channel Coding (JSCC) of text from [19] using recurrent neural networks. In [19], the authors observe that sentences that express the same idea have embeddings that are close together in Hamming distance. But they use cross entropy between words and estimated words as the loss function and use the word error rate as the performance measure, which both do not reflect if two sentences have the same meaning but rather that both are exactly the same.

As a result, semantic communication is still a nascent field; it still remains unclear what this term exactly means [27] and, in particular, its distinction from JSCC [19,28]. As a result, many survey papers aim to provide an interpretation, see, e.g., [9,10,11,12,13]. We will revisit this issue in Section 4.

3. Main Contributions

The main contributions of this article are:

• Motivated by the approach of Bao, Basu et al. [16,17], we adopt the terminus of a semantic source. Inspired by Weaver’s notion, we bring it to the context of communications by considering the complete Markov chain, including semantic source, communications source, transmit signal, communication channel, and received signal in contrast to both [16,17]. Further, we also extend beyond the example of deterministic entailment relations between “models” and “messages” based on propositional logic in [16,17] to probabilistic semantic channels.
• We define the task of semantic communication in the sense that we perform data compression, coding, and transmission of messages observed such that the semantic Random Variable (RV) at a recipient is best preserved. Basically, we implement joint source-channel coding of messages conveying the semantic RV, but not differentiating between Levels A and B. We formulate the semantic communication design either as an Information Maximization or as an Information Bottleneck (IB) optimization problem [29,30,31].

  –

  Although the approach pursued here again leads to an IB problem as in [26], our article introduces a new classification and perspective of semantic communication and different ML-based solution approaches. Different from [26], we solve the IB problem maximizing the mutual information for a fixed encoder output dimension that bounds the information rate.

  –

  The publication presented here differs also both in the interpretation of what is meant by semantic information and in the objective of recovering this semantic information from approaches to semantic communication presented in the literature like, e.g., [21,32].

• Finally, we propose the ML-based semantic communication system SINFONY for a distributed multipoint scenario in contrast to [26]: SINFONY communicates the meaning behind multiple messages that are observed at different senders to a single receiver for semantic recovery. Compared to the distributed scenario in [33,34], we include the communication channel.
• We analyze SINFONY by processing images as an example of messages. Notably, numerical results reveal a tremendous rate-normalized SNR shift up to 20 dB compared to classically designed communication systems.

4. A Framework for Semantics

4.1. Philosophical Considerations

Despite the much-renewed interest, research on semantic communication is still in its infancy and recent work reveals a differing understanding of the word semantics. In this work, we contribute our interpretation. To motivate it, we shortly revisit the research birth hour of communications from a philosophical point of view; its theoretical foundation was laid by Shannon in his landmark paper [1] in 1948.

He stated that “Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem”. In fact, this viewpoint abstracts all kinds of information one may transmit, e.g., oral and written speech, sensor data, etc., and also lays the foundation for the research area of Shannon information theory. Thus, it found its way into many other research areas where data or information are processed, including Artificial Intelligence (AI) and especially its subdomain Machine Learning (ML).

Weaver saw this broad applicability of Shannon’s theory back in 1949. In his comprehensive review of [1], he first states that “there seem to be [communication] problems at three levels” [2] already mentioned in Section 2. These three levels are quoted in recent works, where Level C is oftentimes referred to as goal-oriented communication instead [10].

But we note that, in his concluding section, he then questions this segmentation. He argues for the generality of the theory at Level A for all levels and “that the interrelation of the three levels is so considerable that one’s final conclusion may be that the separation into the three levels is really artificial and undesirable”.

It is important to emphasize that the separation is rather arbitrary. We agree with Weaver’s statement because the most important point that is also the focus herein is the definition of the term semantics, e.g., by Basu et al. [16,17]. Note that the entropy of the semantics is less than or equal to the entropy of the messages. Consequently, we can save information rate by introducing meaning or context. In fact, we are able to add arbitrarily many levels of semantic details to the communication problem and optimize communications for a specific semantic background, e.g., an application or human.

4.2. Semantic System Model

4.2.1. Semantic Source and Channel

Now, we will define our information-theoretic system model of semantic communication. Figure 1 shows the schematic of our model. We assume the existence of a semantic source, described as a hidden target multivariate Random Variable (RV) $\mathbf{z}\in {\mathcal{M}}_z^{N_z\times 1}$ from a domain ${\mathcal{M}}_z$ of dimension $N_z$ distributed according to a probability density or mass function (pdf/pmf) $p\left(\mathbf{z}\right)$. To simplify the discussion, we assume it to be discrete and memoryless. For the remainder of the article, note that the domain of all RVs $\mathcal{M}$ may be either discrete or continuous. Further, we note that the definition of entropy for discrete and continuous RVs differs. For example, the differential entropy of continuous RVs may be negative whereas the entropy of discrete RVs is always positive [35]. Without loss of generality, we will thus assume all RVs either to be discrete or to be continuous. In this work, we avoid notational clutter by using the expected value operator, replacing the integral by summation over discrete RVs, the equations are also valid for discrete RVs and vice versa.

Our approach is similar to that of [16,17]. In [16,17], the semantic source is described by “models of the world”. (Note that, in [17], the semantic information source is defined as a tuple $(\mathbf{z},\mathbf{s},p(\mathbf{z},\mathbf{s}),L)$. In this original notation, $\mathbf{z}$ is the model, $\mathbf{s}$ the message, $p(\mathbf{z},\mathbf{s})$ the joint distribution of $\mathbf{z}$ and $\mathbf{s}$, and L is the deterministic formal language.) In [17], a semantic channel then generates messages through entailment relations between “models” and “messages”. We will call these “messages” source signal and define it to be a RV $\mathbf{s}\in {\mathcal{M}}_s^{N_s\times 1}$ as it is usually observed and enters the communication system. In the classic Shannon design, the aim is to reconstruct the source $\mathbf{s}$ as accurately as possible at the receiver side. Further, we note that the authors in [17] considered the example of a semantic channel with deterministic entailment relations between $\mathbf{z}$ and $\mathbf{s}$ based on propositional logic. In this article, we go beyond this assumption and consider probabilistic semantic channels modeled by distribution $p\left(\mathbf{s}\right|\mathbf{z})$ that include the entailment in [17] as special cases, i.e., $p\left(\mathbf{s}\right|\mathbf{z})=\delta (\mathbf{s}-f\left(\mathbf{z}\right))$ where $\delta (·)$ is the Dirac delta function and $f(·)$ is any generic function. Our viewpoint is motivated by the recent success of pattern recognition tools that advanced the field of AI in the 2010s and may be used to extract semantics [7].

Our approach also extends models as in [21]. There, the authors design a semantic communication system for the transmission of written language/text similar to [19] using transformer networks. In contrast to our work, [21] does not define meaning as RV $\mathbf{z}$. The objective in [21] is to reconstruct $\mathbf{s}$ (sentences) as well as possible, rather than the meaning (RV $\mathbf{z}$) conveyed in $\mathbf{s}$. Optimization is completed with regard to a loss function consisting of two parts, cross entropy between language input $\mathbf{s}$ and output estimate $\widehat{\mathbf{s}}$, as well as a scaled mutual information term between transmit signal $\mathbf{x}$ and receive signal $\mathbf{y}$. After optimization, the authors measure semantic performance by some semantic metric $\mathcal{L}(\mathbf{s},\widehat{\mathbf{s}})$.

We now provide an example to explain what we understand under a semantic source $\mathbf{z}$ and channel $p\left(\mathbf{s}\right|\mathbf{z})$. Let us imagine a biologist who has an image of a tree. The biologist wants to know what kind of tree it is by interpreting the observed data (image). In this case, the semantic source $\mathbf{z}$ is a multivariate RV composed of a categorical RV with M tree classes. For any realization (sample value) ${\mathsf{z}}_i$ of the semantic source, the semantic channel $p\left(\mathbf{s}\right|\mathbf{z})$ then outputs with some probability one image ${\mathsf{s}}_i$ of a tree conveying characteristics of $\mathbf{z}$, i.e., its meaning. Note that the underlying meaning of the same sensed data (message) can be different for other recipients, e.g., humans or tasks/applications, i.e., in other semantic contexts. Imagine a child, i.e., a person with different characteristics (personality, expertise, knowledge, goals, and intentions) than the biologist, who is only interested if he/she can climb up this tree or whether the tree provides shade. Thus, we include the characteristics of the sender and receiver in RV $\mathbf{z}$ and consider it directly in compression and encoding.

Compared to [16], we, therefore, argue that we also include level C by semantic source and channel since context can be included on increasing layers of complexity. First, a RV ${\mathbf{z}}_1$ might capture the interpretation, like the classification of images or sensor data. Moving beyond the first semantic layer, then a RV ${\mathbf{z}}_2$ might expand this towards a more general goal, like keeping a constant temperature in power plant control. In fact, we can add or remove context, i.e., semantics and goals, arbitrarily often according to the human or application behind, and we can optimize the overall (communication) system with regard to ${\mathbf{z}}_1,{\mathbf{z}}_2,\cdots ,{\mathbf{z}}_i$, respectively.

As a last remark, we note that we basically defined probabilistic semantic relationships, and it remains the question of how exactly they might look. In our example, the meaning of the images needs to be labeled into real-world data pairs $\{{\mathsf{s}}_i,{\mathsf{z}}_i\}$ by experts/humans, since image recognition lacks precise mathematical models. This is also true for NLP [21]; how can we measure if two sentences have the same meaning, i.e., how does the semantic space look like? In contrast, in [17], the authors are able to solve their well-defined technical problem (motion detection) by a model-driven approach. We can thus distinguish between model and data-driven semantics, which both can be handled within Shannon’s information theory.

4.2.2. Semantic Channel Encoding

After the semantic source and channel in Figure 1, we extend upon [16] by differentiating between “message”/source signal $\mathbf{s}$ and transmit signal $\mathbf{x}\in {\mathcal{M}}_x^{N_{\mathrm{Tx}}\times 1}$. Our challenge is to encode the source signal $\mathbf{s}$ onto the transmit signal vector $\mathbf{x}$ for reliable semantic communication through the physical communication channel $p\left(\mathbf{y}\right|\mathbf{x})$, where $\mathbf{y}\in {\mathcal{M}}_y^{N_{\mathrm{Rx}}\times 1}$ is the received signal vector. We assume the encoder $p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})$ to be parametrized by a parameter vector $\mathit{\theta }\in {\mathbb{R}}^{N_{\mathit{\theta }}\times 1}$. Note that $p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})$ is probabilistic here, but assumed to be deterministic in communications with $p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})=\delta (\mathbf{x}-{\mu }_{\mathit{\theta }}\left(\mathbf{s}\right))$ and encoder function ${\mu }_{\mathit{\theta }}\left(\mathbf{s}\right)$.

In summary, in contrast to both [16,17], we consider the complete Markov chain $\mathbf{z}\leftrightarrow \mathbf{s}\leftrightarrow \mathbf{x}\leftrightarrow \mathbf{y}$ including semantic source $\mathbf{z}$, communications source $\mathbf{s}$, transmit signal $\mathbf{x}$ and receive signal $\mathbf{y}$. By this means, we distinguish from [17] which only deals with semantic compression, and [16] which is about joint semantic compression and channel coding (Level B). In [16], the authors consider the classic transmission system (Level A) as the (semantic) channel (not to be confused with the definition of the semantic channel in [17] which we make use of in this publication).

At the receiver side, one approach is maximum a posteriori decoding with regard to RV $\mathbf{s}$ that uses the posterior $p_{\mathit{\theta }}\left(\mathbf{s}\right|\mathbf{y})$, being deduced from prior $p\left(\mathbf{s}\right)$ and likelihood $p_{\mathit{\theta }}\left(\mathbf{y}\right|\mathbf{s})$ by application of Bayes law. Based on the estimate of $\mathbf{s}$, then the receiver interprets the actual semantic content $\mathbf{z}$ by $p\left(\mathbf{z}\right|\mathbf{s})$.

Another approach we propose is to include the semantic hidden target RV $\mathbf{z}$ into the design by processing $p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})$. If the calculation of the posterior is intractable, we can replace $p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})$ by the approximation $q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})$, i.e., the semantic decoder, with parameters $\mathit{\phi }\in {\mathbb{R}}^{N_{\mathit{\phi }}\times 1}$. We expect the following benefit: We assume the entropy $\mathcal{H}\left(\mathbf{z}\right)=E_{\mathbf{z}\sim p\left(\mathbf{z}\right)}[-lnp\left(\mathbf{z}\right)]$ of the semantic RV $\mathbf{z}$, i.e., the actual semantic uncertainty or information content, to be less or equal to the entropy $\mathcal{H}\left(\mathbf{s}\right)$ of the source $\mathbf{s}$, i.e., $\mathcal{H}\left(\mathbf{z}\right)\le \mathcal{H}\left(\mathbf{s}\right)$. There, $E_{\mathbf{x}\sim p\left(\mathbf{x}\right)}\left[f\left(\mathbf{x}\right)\right]$ denotes the expected value of $f\left(\mathbf{x}\right)$ with regard to both discrete or continuous RVs $\mathbf{x}$. Consequently, since we would like to preserve the relevant, i.e., semantic, RV $\mathbf{z}$ rather than $\mathbf{s}$, we can compress more s.t. preserving $\mathbf{z}$ conveyed in $\mathbf{s}$. Note that in semantic communication the relevant variable is $\mathbf{z}$, not $\mathbf{s}$. Thus, processing $p_{\mathit{\theta }}\left(\mathbf{s}\right|\mathbf{y})$ without taking $\mathbf{z}$ into consideration resembles the classical approach. Instead of using (and transmitting) $\mathbf{s}$ for inference of $\mathbf{z}$, we now want to find a compressed representation $\mathbf{y}$ of $\mathbf{s}$ containing the relevant information about $\mathbf{z}$.

4.3. Semantic Communication Design via InfoMax Principle

After explaining the system model and the basic components, we are able to approach a semantic communication system design. We first define an optimization problem to obtain the encoder $p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})$ following the Information Maximization (InfoMax) principle from an information theoretic perspective [35]. Thus, we like to find the distribution $p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})$ that maps $\mathbf{s}$ to a representation $\mathbf{x}$ such that most information of the relevant RV $\mathbf{z}$ is included in $\mathbf{y}$, i.e., we maximize the Mutual Information (MI) $I_{}\left(\mathbf{z};\mathbf{y}\right)$ with regard to $p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})$ [36]:

$$\begin{array}{}\hfill \mathrm{(1)}& \underset{p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})}{arg\; max}{I}_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)\hfill \hfill \mathrm{(2)}& =\underset{p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})}{arg\; max}{E}_{\mathbf{z},\mathbf{y}\sim p_{\mathit{\theta }}(\mathbf{z},\mathbf{y})}\left[ln\frac{p_{\mathit{\theta }}(\mathbf{z},\mathbf{y})}{p\left(\mathbf{z}\right)p_{\mathit{\theta }}\left(\mathbf{y}\right)}\right]\hfill \hfill \mathrm{(3)}& =\underset{p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})}{arg\; max}\mathcal{H}\left(\mathbf{z}\right)-\mathcal{H}\left(p_{\mathit{\theta }}(\mathbf{z},\mathbf{y}),p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})\right)\hfill \hfill \mathrm{(4)}& =\underset{p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})}{arg\; max}{E}_{\mathbf{z},\mathbf{y}\sim p_{\mathit{\theta }}(\mathbf{z},\mathbf{y})}\left[ln{p}_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})\right].\hfill \end{array}$$

There, $\mathcal{H}\left(p\left(\mathbf{x}\right),q\left(\mathbf{x}\right)\right)=E_{\mathbf{x}\sim p\left(\mathbf{x}\right)}[-lnq\left(\mathbf{x}\right)]$ is the cross entropy between two pdfs/pmfs $p\left(\mathbf{x}\right)$ and $q\left(\mathbf{x}\right)$. Note independence from $\mathit{\theta }$ in $\mathcal{H}\left(\mathbf{z}\right)$ and dependence in $p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})$ and $p_{\mathit{\theta }}(\mathbf{z},\mathbf{y})$ through the Markov chain $\mathbf{z}\to \mathbf{s}\to \mathbf{y}$. Problem (1) is concave with regard to the encoder $p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})$ for fixed $p\left(\mathbf{s}\right)$ [37], but not necessarily concave with regard to the encoder parameters $\mathit{\theta }$. For example, it is non-concave if the encoder function is non-convex with regard to its parameters being typically the case with DNN encoders. It is worth mentioning that we so far have not set any constraint on the variables we deal with. Hence, the form of $p_{\mathit{\theta }}\left(\mathbf{y}\right|\mathbf{s})$ has to be constrained to avoid learning a trivial identity mapping $\mathbf{y}=\mathbf{s}$. We indeed constrain the optimization by our communication channel $p\left(\mathbf{y}\right|\mathbf{x})$ we assume to be given.

If the calculation of the posterior $p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})$ in (4) is intractable, we are able to replace it by a variational distribution $q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})$ with parameters $\mathit{\phi }$. Similar to the transmitter, DNNs are usually proposed [21,38] for the design of the approximate posterior $q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})$ at the receiver. To improve the performance complexity trade-off, the application of deep unfolding can be considered, a model-driven learning approach that introduces model knowledge of $p_{\mathit{\theta }}(\mathbf{s},\mathbf{x},\mathbf{y},\mathbf{z})$ to create $q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})$ [8,39]. With $q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})$, we are able to define a Mutual Information Lower Bound (MILBO) [36] similar to the well-known Evidence Lower Bound (ELBO) [7]:

$$\begin{array}{}\hfill \mathrm{(5)}& \hfill I_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)& \ge E_{\mathbf{z},\mathbf{y}\sim p_{\mathit{\theta }}(\mathbf{z},\mathbf{y})}\left[ln{q}_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right]\hfill \hfill \mathrm{(6)}& \hfill & =E_{\mathbf{y}\sim p_{\mathit{\theta }}\left(\mathbf{y}\right)}\left[E_{\mathbf{z}\sim p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})}\left[ln{q}_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right]\right]\hfill \hfill \mathrm{(7)}& \hfill & =-E_{\mathbf{y}\sim p_{\mathit{\theta }}\left(\mathbf{y}\right)}\left[\mathcal{H}\left(p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y}),q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right)\right]\hfill \hfill \mathrm{(8)}& \hfill & =-{\mathcal{L}}_{\mathit{\theta },\mathit{\phi }}^{\mathrm{CE}}.\hfill \end{array}$$

The lower bound holds since $-\mathcal{H}\left(p_{\mathit{\theta }}(\mathbf{z},\mathbf{y}),p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})\right)$ itself is a lower bound of the expression in (3) and $E_{\mathbf{z},\mathbf{y}\sim p_{\mathit{\theta }}(\mathbf{z},\mathbf{y})}\left[ln{p}_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})/ln{q}_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right]\ge 0$. Now, we can calculate optimal values of $\mathit{\theta }$ and $\mathit{\phi }$ of our semantic communication design by minimizing the amortized cross entropy ${\mathcal{L}}_{\mathit{\theta },\mathit{\phi }}^{\mathrm{CE}}$ in (7), i.e., marginalized across observations $\mathbf{y}$ [8].

Thus, the idea is to learn parametrizations of the transmitter discriminative model and of the variational receiver posterior, e.g., by AEs or reinforcement learning. Note that, in our semantic problem (1), we do not auto-encode the hidden $\mathbf{z}$ itself, but encode $\mathbf{s}$ to obtain $\mathbf{z}$ by decoding. This can be seen from Figure 1 and by rewriting the amortized cross entropy (7) and (8):

$$\begin{array}{}\hfill \mathrm{(9)}& \hfill {\mathcal{L}}_{\mathit{\theta },\mathit{\phi }}^{\mathrm{CE}}& =E_{\mathbf{y}\sim p\left(\mathbf{y}\right)}\left[\mathcal{H}\left(p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y}),q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right)\right]\hfill \hfill \mathrm{(10)}& \hfill & \begin{array}{c}=E_{\mathbf{s},\mathbf{x},\mathbf{y},\mathbf{z}\sim p_{\mathit{\theta }}(\mathbf{s},\mathbf{x},\mathbf{y},\mathbf{z})}\left[-ln{q}_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right]\hfill \\ =E_{\mathbf{s},\mathbf{z}\sim p(\mathbf{s},\mathbf{z})}\left[E_{\mathbf{x}\sim p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})}\left[E_{\mathbf{y}\sim p\left(\mathbf{y}\right|\mathbf{x})}\left[-ln{q}_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right]\right]\right].\hfill \end{array}\end{array}$$

We can further prove the amortized cross entropy to be decomposable into

$$\begin{array}{}\hfill \mathrm{(11)}& \hfill {\mathcal{L}}_{\mathit{\theta },\mathit{\phi }}^{\mathrm{CE}}& =E_{\mathbf{y}\sim p_{\mathit{\theta }}\left(\mathbf{y}\right)}\left[E_{\mathbf{z}\sim p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})}[-ln{q}_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})+ln{p}_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})-ln{p}_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})]\right]\hfill \hfill \mathrm{(12)}& \hfill & =E_{\mathbf{y}\sim p_{\mathit{\theta }}\left(\mathbf{y}\right)}\left[D_{\mathrm{KL}}\left(p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})\Vert q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right)\right]+\underset{=-I_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)+\mathcal{H}\left(\mathbf{z}\right)}{\underbrace{\mathcal{H}\left(\mathbf{z}|\mathbf{y}\right)}}\hfill \hfill \mathrm{(13)}& \hfill & =\mathcal{H}\left(\mathbf{z}\right)-\underset{\mathrm{enc}. \mathrm{objective}}{\underbrace{I_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)}}+\underset{\mathrm{dec}. \mathrm{objective}}{\underbrace{E_{\mathbf{y}\sim p_{\mathit{\theta }}\left(\mathbf{y}\right)}\left[D_{\mathrm{KL}}\left(p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})\Vert q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right)\right]}}.\hfill \end{array}$$

In the end, maximization of the MILBO with regard to $\mathit{\theta }$ and $\mathit{\phi }$ balances maximization of the mutual information $I_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)$ and minimization of the Kullback–Leibler (KL) divergence $D_{\mathrm{KL}}\left(p_{\mathit{\theta }}\left(\mathbf{z}\right|\mathbf{y})\Vert q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right)$. The former objective can be seen as a regularization term that favors encoders with high mutual information, for which decoders can be learned that are close to the true posterior.

4.4. Classical Design Approach

If we consider classical communication design approaches, we would solve the problem

$$\begin{array}{cc}\hfill & \underset{p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})}{arg\; max}{I}_{}\left(\mathbf{s};\mathbf{y}\right)\hfill \end{array}$$

(14)

which relates to Joint Source-Channel Coding (JSCC). There, the aim is to find a representation $\mathbf{x}$ that retains a significant amount of information about the source signal $\mathbf{s}$ in $\mathbf{y}$. Again, we can apply the lower bound (8). In fact, bounding (14) by (8) shows that approximate maximization of the mutual information justifies the minimization of the cross entropy in the AutoEncoder (AE) approach [6], often seen in recent wireless communication literature [6,19,28].

4.5. Information Bottleneck View

It should be stressed that we have not set any constraints on the variables in the InfoMax problem so far. However, in many applications, compression is needed because of the limited information rate. Therefore, we can formulate an optimization problem where we like to maximize the relevant information $I_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)$ subject to the constraint to limit the compression rate $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)$ to a maximum information rate $I_{\mathrm{C}}$:

$$\begin{array}{cc}\hfill & \underset{p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})}{arg\; max}{I}_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)\mathrm{s}.\mathrm{t}.I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)\le I_{\mathrm{C}}.\hfill \end{array}$$

(15)

Problem (15) is an important variation of the InfoMax principle and called the Information Bottleneck (IB) problem [10,29,40,41]. The IB method introduced by Tishby et al. [29] has been the subject of intensive research for years and has proven to be a suitable mathematical/information-theoretical framework for solving numerous problems—as well as in wireless communications [30,31,42,43]. Note that we aim for an encoder that compresses $\mathbf{s}$ into a compact representation $\mathbf{x}$ for discrete RVs by clustering and for continuous RVs by dimensionality reduction.

To solve the constrained optimization problem (15), we can use Lagrangian optimization and obtain

$$\begin{array}{cc}\hfill & \underset{p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})}{arg\; max}{I}_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)-\beta I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)\hfill \end{array}$$

(16)

with Lagrange multiplier $\beta \ge 0$. The Lagrange multiplier $\beta$ allows the defining of a trade-off between the relevant information $I_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)$ and compression rate $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)$, which indicates the relation to rate distortion theory [30]. With $\beta =0$, we have the InfoMax problem (1) whereas for $\beta \to \infty$ we minimize compression rate. Calculation of the mutual information terms may be computationally intractable, as in the InfoMax problem (1). Approximation approaches can be found in [44,45]. Notable exceptions include if the RVs are all discrete or Gaussian distributed.

We note that in [10,26] the authors already introduced the IB problem to task-oriented communications. But [10,26] do not address our viewpoint or classification. We compress and channel encode the messages/communications source $\mathbf{s}$ for given entailment $p\left(\mathbf{s}\right|\mathbf{z})$, in the sense of a data-reduced and reliable communication of the semantic RV $\mathbf{z}$. Basically, we implement joint source-channel coding of $\mathbf{s}$ s.t. preserving the semantic RV $\mathbf{z}$, and we do not differentiate between Levels A and B, as indicated by Weaver’s notion outlined in Section 2. Indeed, we draw a direct connection to IB compared to related semantic communication literature [19,21,38] that, so far, only included optimization with terms reminiscent of the IB problem.

4.5.1. Semantic Information Bottleneck

This article does not only distinct itself on a conceptual, but also on a technical level from [26,34]. We follow a different strategy to solve (15).

First, using the data processing inequality [46], we see that the compression rate is upper bounded by the mutual information of the encoder $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{x}\right)$ and that of the channel $I_{}\left(\mathbf{x};\mathbf{y}\right)$:

$$\begin{array}{c}\hfill I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)\le min\left\{I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{x}\right),I_{}\left(\mathbf{x};\mathbf{y}\right)\right\}.\end{array}$$

(17)

In case of negligible encoder compression $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{x}\right)>I_{}\left(\mathbf{x};\mathbf{y}\right)$, the channel becomes the limiting factor of information rate. For example, with a deterministic continuous mapping $\mathbf{x}={\mu }_{\mathit{\theta }}\left(\mathbf{s}\right)$, this is true since $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{x}\right)\to \infty$. Using the chain rule of mutual information [46], we see that this upper bound on compression rate grows with the dimension of $\mathbf{x}$, i.e., the number of channel uses $N_{\mathrm{Tx}}$:

$$\begin{array}{c}\hfill I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)\le I_{}\left(\mathbf{x};\mathbf{y}\right)=\sum _{n=1}^{N_{\mathrm{Tx}}}\underset{\ge 0}{\underbrace{I_{}\left(x_n;\mathbf{y}|x_{n-1},\cdots ,x_1\right)}}.\end{array}$$

(18)

Assuming $\mathbf{y}$ to be conditional dependent on $x_n$ given $x_{n-1},\cdots ,x_1$, i.e., $p\left(\mathbf{y}\right|x_n,\cdots ,x_1)\ne p\left(\mathbf{y}\right|x_{n-1},\cdots ,x_1)$ being, e.g., true for an AWGN channel, it is $I_{}\left(x_n;\mathbf{y}|x_{n-1},\cdots ,x_1\right)>0$ [46] and the sum in (18) indeed strictly increases. Replacing $\mathbf{y}$ in $I_{}\left(\mathbf{x};\mathbf{y}\right)$ of (18) by $\mathbf{s}$, the result also holds for encoder compression $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{x}\right)$, respectively. Hence, increasing the encoder output dimension $N_{\mathrm{Tx}}$, we can increase the possible compression rate $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)$. Interchanging $\mathbf{x}$ and $\mathbf{y}$ in (18), we see that the same holds for the receiver input dimension $N_{\mathrm{Rx}}$.

Furthermore, the mutual information of the channel and, thus, the compression rate are upper bounded by channel capacity:

$$\begin{array}{c}\hfill I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)\le I_{}\left(\mathbf{x};\mathbf{y}\right)\le \underset{p\left(\mathbf{x}\right);E_{}\left[\left|\right[|x_n{\left|\right]}^2\right]\le 1}{max}{I}_{}\left(\mathbf{x};\mathbf{y}\right)=C.\end{array}$$

(19)

For example, with an AWGN channel with noise standard deviation ${\sigma }_{\mathrm{n}}$, we have $C=N_{\mathrm{Tx}}/2·ln\left(1+1/{\sigma }_{\mathrm{n}}^2\right)$ again increasing with $N_{\mathrm{Tx}}$.

Now, let us assume the RVs to be discrete so that $\mathcal{H}\left(\mathbf{x}|\mathbf{s}\right)\ge 0$. Indeed, this is true if the RVs are processed discretely with finite resolution on digital signal processors, as in the numerical example of Section 5. As long as $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{x}\right)<C$, all information of the discrete RVs can be transmitted through the channel with arbitrary low error probability according to Shannon’s channel coding theorem [1]. Then, we can upper bound encoder compression $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{x}\right)$ and thus compression rate $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)$ by the sum of entropies of any output $x_n$ [46] of the encoder $p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})$—each with cardinality $|{\mathcal{M}}_x|$:

$$I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{x}\right)=\mathcal{H}\left(\mathbf{x}\right)-\underset{\ge 0}{\underbrace{\mathcal{H}\left(\mathbf{x}|\mathbf{s}\right)}}\le \mathcal{H}\left(\mathbf{x}\right)\le \sum _{n=1}^{N_{\mathrm{Tx}}}\mathcal{H}\left(x_n\right)\le N_{\mathrm{Tx}}·{log}_2\left(\right|{\mathcal{M}}_x\left|\right).$$

(20)

Note that the entropy sum in (20) grows again with $N_{\mathrm{Tx}}$ for discrete RVs since $0\le \mathcal{H}\left(x_n\right)\le {log}_2\left(\right|{\mathcal{M}}_x\left|\right)$. Moreover, we can define an encoder capacity $C_{\mathit{\theta }}$ analogous to channel capacity C in (19) that upper bounds encoder compression $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{x}\right)$. It may be restricted by the chosen (DNN) model $p_{\mathit{\theta }}\left(\mathbf{x}\right|\mathbf{s})$ and optimization procedure with regard to $\mathit{\theta }$, i.e., the hypothesis class [7].

In summary, we have proven by (19) and (20) that there is an information bottleneck when maximizing the relevant information $I_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)$ either due to the channel distortion $I_{}\left(\mathbf{x};\mathbf{y}\right)$ or encoder compression $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)$.

To fully exploit the available resources, we set constraint $I_{\mathrm{C}}$ to be equal to the upper bound, i.e., channel capacity C or the upper bound on encoder compression rate $N_{\mathrm{Tx}}·{log}_2\left(\right|{\mathcal{M}}_x\left|\right)$. In both cases, the upper bound grows (linearly) with the encoder output dimension $N_{\mathrm{Tx}}$, and, thus, we can set the constraint $I_{\mathrm{C}}$ higher or lower by choosing $N_{\mathrm{Tx}}$.

With fixed constraint $I_{\mathrm{C}}$, we maximize the relevant information $I_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)$. By doing so, we derive an exact solution to (15) that maximizes $I_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)$ for a fixed encoder output dimension that bounds the compression rate. As in the InfoMax problem, we can exploit the MILBO to use the amortized cross entropy ${\mathcal{L}}_{\mathit{\theta },\mathit{\phi }}^{\mathrm{CE}}$ in (9) as the optimization criterion.

4.5.2. Variational Information Bottleneck

In [26], however, the authors solve the variational IB problem of (16) and require tuning of $\beta$. Albeit also using the MILBO as a variational approximation to the first term in (16), they introduce a KL divergence term as an upper bound to compression rate $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)$ derived by $D_{\mathrm{KL}}\left(p_{\mathit{\theta }}\left(\mathbf{y}\right)\Vert q_{\mathit{\vartheta }}\left(\mathbf{y}\right)\right)\ge 0$ with some variational distribution $q_{\mathit{\vartheta }}\left(\mathbf{y}\right)$ with parameters $\mathit{\vartheta }$ [44]. Then, the variational IB objective function reads [44]:

$$\begin{array}{cc}\hfill & I_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)-\beta I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)\ge E_{\mathbf{z},\mathbf{y}\sim p_{\mathit{\theta }}(\mathbf{z},\mathbf{y})}\left[ln{q}_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right]-\beta ·E_{\mathbf{s}\sim p\left(\mathbf{s}\right)}\left[D_{\mathrm{KL}}\left(p_{\mathit{\theta }}\left(\mathbf{y}\right|\mathbf{s})\Vert q_{\mathit{\vartheta }}\left(\mathbf{y}\right)\right)\right].\hfill \end{array}$$

(21)

Moreover, the authors use a log-uniform distribution as the variational prior $q_{\mathit{\vartheta }}\left(\mathbf{y}\right)$ in [26] to induce sparsity on $\mathbf{y}$ so that the number of outputs is dynamically determined based on the channel condition or SNR, i.e., $p_{\mathit{\theta }}\left(\mathbf{y}\right|\mathbf{s},{\sigma }_{\mathrm{n}}^2)$. The approach additionally necessitates approximation of the KL divergence term in (21) and estimation of the noise variance ${\sigma }_{\mathrm{n}}^2$.

With our approach we avoid the additional approximations and tuning of the hyperparameter $\beta$ in (21) possibly enabling better semantic performance as well as reduced inference and training complexity at the cost of full usage of $N_{\mathrm{Tx}}$ channels even when the channel capacity C enables its reduction. We leave a numerical comparison to [26] for future research as this is out of the scope of this paper.

4.6. Implementation Considerations

Now, we will provide important implementation considerations for optimization of (8)/(10) and (15). We note that computation of the MILBO leads to similar problems as for the ELBO [35]; if calculating the expected value in (10) cannot be solved analytically or is computationally intractable—as typically the case with DNNs—we can approximate it using Monte Carlo sampling techniques with N samples ${\left\{{\mathsf{z}}_i,{\mathsf{s}}_i,{\mathsf{x}}_i,{\mathsf{y}}_i\right\}}_{i=1}^N$.

For Stochastic Gradient Descent (SGD)-based optimization like, e.g., in the AE approach, the gradient with regard to $\mathit{\phi }$ can then be calculated by

$$\begin{array}{}\hfill \mathrm{(22)}& \hfill \frac{\partial {\mathcal{L}}_{\mathit{\theta },\mathit{\phi }}^{\mathrm{CE}}}{\partial \mathit{\phi }}& =\frac{\partial }{\partial \mathit{\phi }}{E}_{\mathbf{z},\mathbf{s},\mathbf{y}\sim p_{\mathit{\theta }}\left(\mathbf{y}\right|\mathbf{s})p\left(\mathbf{s}\right|\mathbf{z})p\left(\mathbf{z}\right)}\left[-ln{q}_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right]\hfill \hfill \mathrm{(23)}& \hfill & =-E_{\mathbf{z},\mathbf{s},\mathbf{y}\sim p_{\mathit{\theta }}\left(\mathbf{y}\right|\mathbf{s})p\left(\mathbf{s}\right|\mathbf{z})p\left(\mathbf{z}\right)}\left[\frac{\partial ln{q}_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})}{\partial \mathit{\phi }}\right]\hfill \hfill \mathrm{(24)}& \hfill & \approx -\frac{1}{N}\sum _{i=1}^N\frac{\partial ln{q}_{\mathit{\phi }}\left({\mathsf{z}}_i\right|{\mathsf{y}}_i)}{\partial \mathit{\phi }}\hfill \end{array}$$

with N being equal to the batch size $N_{\mathrm{b}}$ and by application of the backpropagation algorithm to $\frac{\partial }{\partial \mathit{\phi }}ln{q}_{\mathit{\phi }}\left({\mathsf{z}}_i\right|{\mathsf{y}}_i)=\frac{\partial }{\partial \mathit{\phi }}{q}_{\mathit{\phi }}\left({\mathsf{z}}_i\right|{\mathsf{y}}_i)/q_{\mathit{\phi }}\left({\mathsf{z}}_i\right|{\mathsf{y}}_i)$ in Automatic Differentiation Frameworks (ADF), e.g., TensorFlow and PyTorch. Computation of the so-called REINFORCE gradient with regard to $\mathit{\theta }$ leads to a high variance of the gradient estimate since we sample with regard to the distribution $p_{\mathit{\theta }}\left(\mathbf{y}\right|\mathbf{s})$ dependent on $\mathit{\theta }$ [35].

Reparametrization Trick

Leveraging the direct relationship between $\mathit{\theta }$ and $\mathbf{y}$ in $ln{q}_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})$ can help reduce the estimator’s high variance. Typically, e.g., in the Variational AE (VAE) approach, the reparametrization trick is used to achieve this [35]. Here, we can apply it if we can decompose the latent variable $\mathbf{y}\sim p_{\mathit{\theta }}\left(\mathbf{y}\right|\mathbf{s})$ into a differentiable function $\mathbf{y}=f_{\mathit{\theta }}(\mathbf{s},\mathbf{n})$ and a RV $\mathbf{n}\sim p\left(\mathbf{n}\right)$ independent of $\mathit{\theta }$. Fortunately, the typical forward model of a communication system $p_{\mathit{\theta }}\left(\mathbf{y}\right|\mathbf{s})$ fulfills this criterion. Assuming a deterministic DNN encoder $\mathbf{x}={\mu }_{\mathit{\theta }}\left(\mathbf{s}\right)$ and additive noise $\mathbf{n}$ with covariance $\mathsf{\Sigma }$, we can thus rewrite $\mathbf{y}$ into $f_{\mathit{\theta }}(\mathbf{s},\mathbf{n})={\mu }_{\mathit{\theta }}\left(\mathbf{s}\right)+{\mathsf{\Sigma }}^{1/2}·\mathbf{n}$ and, accordingly, the amortized cross entropy gradient into:

$$\begin{array}{}\hfill \mathrm{(25)}& \hfill \frac{\partial {\mathcal{L}}_{\mathit{\theta },\mathit{\phi }}^{\mathrm{CE}}}{\partial \mathit{\theta }}& =-\frac{\partial }{\partial \mathit{\theta }}{E}_{\mathbf{z},\mathbf{s},\mathbf{y}\sim p_{\mathit{\theta }}\left(\mathbf{y}\right|\mathbf{s})p(\mathbf{s},\mathbf{z})}\left[ln{q}_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})\right]\hfill \hfill \mathrm{(26)}& \hfill & =-E_{\mathbf{z},\mathbf{s},\mathbf{n}\sim p\left(\mathbf{n}\right)p\left(\mathbf{s}\right|\mathbf{z}\left)p\right(\mathbf{z})}\left[\frac{\partial f_{\mathit{\theta }}(\mathbf{s},\mathbf{n})}{\partial \mathit{\theta }}·\frac{\partial ln{q}_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})}{\partial \mathbf{y}}\right]\hfill \hfill \mathrm{(27)}& \hfill & \approx -\frac{1}{N}\sum _{i=1}^N\frac{\partial f_{\mathit{\theta }}({\mathsf{s}}_i,{\mathsf{n}}_i)}{\partial \mathit{\theta }}·\frac{\partial ln{q}_{\mathit{\phi }}\left({\mathsf{z}}_i\right|{\mathsf{y}}_i)}{\partial \mathbf{y}}{|}_{\mathbf{y}=f_{\mathit{\theta }}({\mathsf{s}}_i,{\mathsf{n}}_i)}.\hfill \end{array}$$

The reparametrization trick can be easily implemented in ADFs by adding a noise layer—typically used for regularization in ML literature—after (DNN) function $\mathbf{x}={\mu }_{\mathit{\theta }}\left(\mathbf{s}\right)$. Then, our loss function (10) amounts to

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathit{\theta },\mathit{\phi }}^{\mathrm{CE}}\approx -\frac{1}{N}\sum _{i=1}^{N}ln{q}_{\mathit{\phi }}\left({\mathsf{z}}_i\right|{\mathbf{y}}_i=f_{\mathit{\theta }}({\mathsf{s}}_i,{\mathsf{n}}_i)).\end{array}$$

(28)

This enables the joint optimization of both $\mathit{\theta }$ and $\mathit{\phi }$, as demonstrated in recent works [6], treating unsupervised optimization of AEs as a supervised learning problem.

5. Example of Semantic Information Recovery

In this section, we provide one numerical example of data-driven semantics to explain what we understand under a semantic communication design and to show its benefits: It is the task of image classification. In fact, we consider our example of the biologist from Section 4.2 who wants to know what type of tree it is.

For the remainder of this article, we will thus assume the hidden semantic RV to be a one-hot vector $\mathbf{z}\in {\{0,1\}}^{M\times 1}$ where all elements are zero except for one element representing one of the M image classes. Then, the semantic channel $p\left(\mathbf{s}\right|\mathbf{z})$ (see Figure 1) generates images belonging to this class, i.e., the source signal $\mathbf{s}$.

Note that for point-to-point transmission, as in [26], we could first classify the image based on the posterior $q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{s})$, as shown in Figure 2 and transmit the estimate $\widehat{\mathbf{z}}$ (encoded into $\mathbf{x}$) through the physical channel since this would be most rate or bandwidth efficient.

But if the image information is distributed across multiple agents, all (sub) images may contribute useful information for classification. We could thus lose information when making hard decisions on each transmitter’s side. In the distributed setting, transmission and combination of features, i.e., soft information, is crucial to obtain high classification accuracy.

Further, we note that transmission of full information, i.e., raw image data $\mathbf{s}$, through a wireless channel from each agent to a central unit for full image classification would consume a lot of bandwidth. This case is also shown in Figure 2 assuming perfect communication links between the output of the semantic channel and the input of the ResNet Feature Extractor.

Therefore, we investigate a distributed setting shown in Figure 3. There, each of four agents sees its own image ${\mathbf{s}}_1,\cdots ,{\mathbf{s}}_4\sim p\left({\mathbf{s}}_i\right|\mathbf{z})$ being generated by the same semantic RV $\mathbf{z}$. Based on these images, a central unit shall extract semantics, i.e., perform classification. We propose to optimize the four encoders $p_{{\mathit{\theta }}_i}\left({\mathbf{x}}_i\right|{\mathbf{s}}_i)$ with $i=1,\cdots ,4$, each consisting of a bandwidth efficient feature extractor (ResNet Feature Extractor i) and transmitter (Tx i) jointly with a decoder $q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y}={[{\mathbf{y}}_1,{\mathbf{y}}_2,{\mathbf{y}}_3,{\mathbf{y}}_4]}^T)$, consisting of a receiver (Rx) and concluding classifier (Classifier), with regard to cross entropy (10) of the semantic labels (see Figure 3). Hence, we maximize the system’s overall semantic measure, i.e., classification accuracy. Note that this scenario is different from both [33,34]; we include a physical communication channel (Comm. Channel i) since we aim to transmit and not only compress. For the sake of simplicity, we assume orthogonal channel access. The IB is addressed by limiting the number of channel uses, which defines the constraint $I_{\mathrm{C}}$ in (15).

As a first demonstration example, we use the grayscale MNIST and colored CIFAR10 datasets with $M=10$ image classes [47]. We assume that the semantic channel generates an image that we divide into four equally sized quadrants and each agent observes one quadrant ${\mathbf{s}}_1,\cdots ,{\mathbf{s}}_4\in {\mathbb{R}}^{N_{\mathrm{x}}\times N_{\mathrm{y}}\times N_{\mathrm{c}}}$, where $N_{\mathrm{x}}$ and $N_{\mathrm{y}}$ is the number of image pixels in the x- and y-dimension, respectively, and $N_{\mathrm{c}}$ is the number of color channels. Albeit this does not resemble a realistic scenario, note that we can still show the basic working principle and ease implementation.

5.1. ResNet

For the design of the overall system, we rely on a famous DNN approach for feature extraction, breaking records at the time of invention: ResNet [47,48]. The key idea of ResNet is that it consists of multiple residual units. Each unit’s input is fed directly to its output and if the dimensions do not match, a convolutional layer is used. This structure enables fast training and convergence of DNNs since the training error can be backpropagated to early layers through these skip connections. From a mathematical point of view, usual DNNs have the design flaw that using a larger function class, i.e., more DNN layers, does not necessarily increase the expressive power. However, this holds for nested functions like ResNet which contain the smaller classes of early layers.

Each residual unit itself consists of two Convolutional NNs (CNNs) with subsequent batch normalization and ReLU activation function, i.e., ${\rho }_1\left(·\right)=max(·,0)$, to extract translation invariant and local features across two spatial dimensions $N_{\mathrm{x}}$ and $N_{\mathrm{y}}$. Color channels, like in CIFAR10, add a third dimension $N_{\mathrm{c}}=3$ and additional information. The idea behind stacking multiple layers of CNNs is that features tend to become more abstract from early layers (e.g., edges and circles) to final layers (e.g., beaks or tires).

In this work, we use the preactivation version of ResNet without bottlenecks from [47,48] implemented for classification on the dataset CIFAR10. In

Table 1. Semantic INFOrmation traNsmission and recoverY (SINFONY)—DNN architecture for distributed image classification.

Component	Layer	Dimension
Input	Image (MNIST, CIFAR10)	$(14,14,1)$, $(16,16,3)$
$4\times$	Conv2D	$(14,14,14)$, $(16,16,16)$
Feature	ResNetBlock (2/3 res. un.)	$(14,14,14)$, $(16,16,16)$
Extractor	ResNetBlock (2/3 res. un.)	$(7,7,28)$, $(8,8,32)$
	ResNetBlock (2/3 res. un.)	$(4,4,56)$, $(4,4,64)$
	Batch Normalization	$(4,4,56)$, $(4,4,64)$
	ReLU activation	$(4,4,56)$, $(4,4,64)$
	GlobalAvgPool2D	$\left(56\right)$, $\left(64\right)$
$4\times$ Tx	ReLU	$N_{\mathrm{Tx}}$
	Linear	$N_{\mathrm{Tx}}$
	Normalization (dim.)	$N_{\mathrm{Tx}}$
$4\times$ Channel	AWGN	$N_{\mathrm{Tx}}$
Rx	ReLU ($4\times$ shared)	$(2,2,N_{\mathrm{w}})$
	GlobalAvgPool2D	$N_{\mathrm{w}}$
Classifier	Softmax	$M=10$

, we show its structure for the distributed scenario from Figure 3. There, ResNetBlock is the basic building block of the ResNet architecture. Each block consists of multiple residual units (res. un.) and we use 2 for the MNIST dataset and 3 for the CIFAR10 dataset, which means we use ResNet14 and ResNet20, respectively. We arrive at the architecture of central image processing from Figure 2 by removing the components Tx, (physical) Channel, and Rx and increasing each spatial dimension by 2 to contain all quadrants of the original image. For further implementation details, we refer the reader to the original work [48].

5.2. Distributed Semantic Communication Design Approach

Our key idea here is to modify ResNet with regard to the communication task by splitting it at a suitable point where a representation $\mathbf{r}\in {\mathbb{R}}^{N_{\mathrm{Feat}}\times 1}$ of semantic information with low-bandwidth is present (see Figure 2 and Figure 3). ResNet and CNNs in general can be interpreted to extract features; with full images, we obtain a feature map of size $8\times 8\times N_{\mathrm{Feat}}$ after the last ReLU activation (see Table 1). These local features are aggregated by the global average pooling layers across the 2 spatial dimensions into $\mathbf{r}$. Based on these $N_{\mathrm{Feat}}$ global features in $\mathbf{r}$, the softmax layer finally classifies the image. We note that the features contain the relevant information with regard to the semantic RV $\mathbf{z}$ and are of low dimension compared to the original image or even its sub-images, i.e., 64 compared to $16\times 16\times 3=768$ for CIFAR10.

Therefore, we aim to transmit each agent’s local features ${\mathbf{r}}_i\in {\mathbb{R}}^{N_{\mathrm{Feat}}\times 1}$ ($i=1,\cdots ,4$) instead of all sub-images ${\mathbf{s}}_i$ and add the component Tx in Table 1 to encode the features ${\mathbf{r}}_i$ into ${\mathbf{x}}_i\in {\mathbb{R}}^{N_{\mathrm{Tx}}\times 1}$ for transmission through the wireless channel (see Figure 3). We note that ${\mathbf{x}}_i\in {\mathbb{R}}^{N_{\mathrm{Tx}}\times 1}$ is analog and that the output dimension $N_{\mathrm{Tx}}$ of ${\mathbf{x}}_i$ defines the number of channel uses per agent/image. Note that the less often we use the wireless channel ($N_{\mathrm{Tx}}$), the less information we transmit but the less bandwidth we consume, and vice versa. Hence, the number of channel uses defines the IB in (15). We implement the Tx module by DNN layers. To limit the transmission power to one, we constrain the Tx output by the norm along the training batch or the encoding vector dimension (dim.), i.e., $x_n={\tilde{x}}_n/\sqrt{E_{}\left[{\tilde{x}}_n^2\right]}$ or ${\mathbf{x}}_i=\sqrt{N_{\mathrm{Tx}}}·{\tilde{\mathbf{x}}}_i/{∥{\tilde{\mathbf{x}}}_i∥}_2$ where ${\tilde{\mathbf{x}}}_i\in {\mathbb{R}}^{N_{\mathrm{Tx}}\times 1}$ is the output of the layer Linear from Table 1. For numerical simulations, we choose all Tx layers to have width $N_{\mathrm{Tx}}$.

At the receiver side, we use a single Rx module only with shared DNN layers and parameters ${\mathit{\phi }}_{\mathrm{Rx}}$ for all inputs ${\mathbf{y}}_i$. This setting would be optimal if any feature is reflected in any sub-image and if the statistics of the physical channels are the same. Exploiting the prior knowledge of location-invariant features and assuming Additive White Gaussian Noise (AWGN) channels, this design choice seems reasonable. In our experiments, all layers of the Rx module have width $N_{\mathrm{w}}$. A larger layer width $N_{\mathrm{w}}$ is equivalent to more computing power.

The output of the Rx module can be interpreted as a representation of the image features ${\mathbf{r}}_i$ with index i indicating the spatial location. Thus, we have a representation of a feature map of size $(2,2,N_{\mathrm{w}})$ that we aggregate across the spatial dimension according to the ResNet structure. Based on this semantic representation, a softmax layer with 10 units finally computes class probabilities $q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{y})$ whose maximum is the maximum a posteriori estimate $\widehat{z}$. In the following, we name our proposed approach Semantic INFOrmation traNsmission and recoverY (SINFONY).

5.3. Optimization Details

We evaluate SINFONY in TensorFlow 2 [49] on the MNIST and CIFAR10 datasets. The source code is available in [50] and the default simulation and training parameters are summarized in

Table 2. Default simulation and training parameters.

Parameter Name	Variable	Value (MNIST, CIFAR10)
Batch size	$N_{\mathrm{b}}$	64
Epoch number	$N_{\mathrm{e}}$	20, 200
Learning rate	$ϵ$	Schedule: $ϵ=\left\{0.1,0.01,0.001\right\}$
		with $N_{\mathrm{e}}=\left\{3,6\right\}$, $\left\{100,150\right\}$
Optimizer		SGD with momentum $=0.9$
Preprocessing		Input normalization to $[0,1]$
Training SNR range	${\mathrm{SNR}}_{\mathrm{train}}$	$[-4,6]$ dB
Training dataset size	$N_{\mathrm{train}}$	60 k, 50 k
Validation dataset size		10 k
Weight decay		$0.0001$
Weight initialization		Glorot uniform, ReLU: He uniform
Encoder normalization	dim.	Batch dimension
Rx layer width	$N_{\mathrm{w}}$	56, 64

. We split the dataset into $N_{\mathrm{train}}=60$ k/50 k training data and 10 k validation data samples, respectively. For preprocessing, we normalize the pixel inputs to range $[0,1]$, but we do not use data augmentation, in contrast to [47,48], yielding slightly worse accuracy. The ReLU layers are initialized with uniform distribution according to He and all other layers according to Glorot [51].

In the case of CIFAR10 classification with central image processing and original ResNet, we need to train $N_{\mathit{\theta }}+N_{\mathit{\phi }}$ = 273,066 parameters. We like to stress that although we divided the image input into four smaller pieces, this number grows more than four times to $4N_{\mathit{\theta }}+N_{\mathit{\phi }}$ = 1,127,754 with $N_{\mathrm{Tx}}=N_{\mathrm{Feat}}=64$ for SINFONY. The reason lies in the ResNet structure with minor dependence on the input image size and that we process at four agents with an additional Tx module. Only $N_{\mathit{\phi }}=4810$ parameters amount to the Rx module and classification, i.e., the central unit. We note that the number of added Tx and Rx parameters of 33,560 and 3192 is relatively small. Since the number of parameters only weakly grows with Rx layer width $N_{\mathrm{w}}$ in our design, we choose $N_{\mathrm{w}}=N_{\mathrm{Feat}}$ as the default.

For optimization of the cross entropy (10) or the loss function (28), we use the reparametrization trick from Section 4.6 and SGD with a momentum of $0.9$ and a batch size of $N_{\mathrm{b}}=64$. We add $l_2$-regularization with a weight decay of $0.0001$ as in [47,48]. The learning rate of $ϵ=0.1$ is reduced to $0.01$ and $0.001$ after $N_{\mathrm{e}}=100$ and 150 epochs for CIFAR10 and after 3 and 6 epochs for MNIST. In total, we train for $N_{\mathrm{e}}=200$ epochs with CIFAR10 and for 20 with MNIST. In order to optimize the transceiver for a wider SNR range, we choose the training SNR to be uniformly distributed within ${\mathrm{SNR}}_{\mathrm{train}}\in [-4,6]$ dB where $\mathrm{SNR}=1/{\sigma }_{\mathrm{n}}^2$ with noise variance ${\sigma }_{\mathrm{n}}^2$.

5.4. Numerical Results and Discussion

In the following, we will investigate the influence of specific design choices on our semantic approach SINFONY. Then, we compare a semantic with a classical Shannon-based transmission approach. The design choices are as follows:

• Central: Central and joint processing of full image information by the ResNet classifier, see Figure 2. It indicates the maximum achievable accuracy.
• SINFONY - Perfect com.: The proposed distributed design SINFONY trained with perfect communication links and without channel encoding, i.e., Tx and Rx module, but with Tx normalization layer. Thus, the plain and power-constrained features are transmitted with $N_{\mathrm{Tx}}=N_{\mathrm{Feat}}$ channel uses. It serves as the benchmark since it indicates the maximum performance of the distributed design.
• SINFONY - AWGN: SINFONY Perfect com. evaluated with AWGN channel.
• SINFONY - AWGN + training: SINFONY Perfect com. trained with AWGN channel.
• SINFONY - Tx/Rx (${\mathbf{N}}_{\mathrm{Tx}}={\mathbf{N}}_{\mathrm{Feat}}$): SINFONY trained with channel encoding, i.e., Tx and Rx module, and $N_{\mathrm{Tx}}=N_{\mathrm{Feat}}$ channel uses.
• SINFONY - Tx/Rx (${\mathbf{N}}_{\mathrm{Tx}}<{\mathbf{N}}_{\mathrm{Feat}}$): SINFONY trained with channel encoding and $N_{\mathrm{Tx}}<N_{\mathrm{Feat}}$ channel uses for feature compression.
• SINFONY - Classic digital com.: SINFONY - Perfect com. with classic digital communications (Huffman coding, LDPC coding with belief propagation decoding, and digital modulation) as additional Tx and Rx modules. For details, see Section 5.4.4.
• SINFONY - Analog semantic AE: SINFONY - Perfect com. with ML-based analog communications (AE with regard to $\mathbf{r}$) as additional Tx and Rx modules. It is basically the semantic communication approach from [19,21,28,32]. For details, see Section 5.4.5.

Since meaning is expressed by the RV $\mathbf{z}$, we use classification accuracy to measure semantic transmission quality. For illustration in logarithmic scale, we show the opposite of accuracy in all plots, i.e., classification error rate.

5.4.1. MNIST Dataset

The numerical results of our proposed approach SINFONY on the MNIST validation dataset are shown in Figure 4 for $N_{\mathrm{w}}=56$. To obtain a fair comparison between transmit signals ${\mathbf{x}}_i\in {\mathbb{R}}^{N_{\mathrm{Tx}}\times 1}$ of different length $N_{\mathrm{Tx}}$, we normalize the SNR by the spectral efficiency or rate $\eta =N_{\mathrm{Feat}}/N_{\mathrm{Tx}}$. First, we observe that the classification error rate of $0.5\%$ of the central ResNet unit with full image information (Central) is smaller than that of $0.9\%$ of SINFONY - Perfect com. Note that we assume ideal communication links. However, the difference seems negligible considering that the local agents only see a quarter of the full images and learn features independently based on it.

With noisy communication links (SINFONY - AWGN), the performance degrades especially for $\mathrm{SNR}<10$ dB, and we can avoid degradation just partly by training with noise (SINFONY - AWGN + training). Introducing the Tx module (SINFONY - Tx/Rx $N_{\mathrm{Tx}}=56$), we further improve classification accuracy at low SNR. If we encode the features from $N_{\mathrm{Feat}}=56$ to only $N_{\mathrm{Tx}}=14$ in the Tx module (SINFONY - Tx/Rx $N_{\mathrm{Tx}}=14$) to have less channel uses/bandwidth (stronger bottleneck), the error rate is lowest compared to other SINFONY examples with non-ideal links for low normalized SNR. At high SNR, we observe a small error offset, which indicates lossy compression. In fact, our system SINFONY learns a reliable semantic encoding to improve the classification performance of the overall system with non-ideal links. Every design choice in Table 1 is well-motivated.

5.4.2. CIFAR10 Dataset

Comparing these results to the classification accuracy on CIFAR10 shown in Figure 5, we observe a similar behavior. But a few main differences become apparent. Central performs much better with a $12\%$ error rate than SINFONY - Perfect com. with $20\%$. We expect the reason to lie in the more challenging dataset with more color channels. Further, SINFONY - AWGN + training with $N_{\mathrm{Tx}}=N_{\mathrm{Feat}}=64$ channel uses runs into a rather high error floor. Notably, even SINFONY - Tx/Rx ($N_{\mathrm{Tx}}=16$) with fewer channel uses performs better than both SINFONY - AWGN and SINFONY - AWGN + training over the whole SNR range and achieves channel encoding with negligible loss. This means adding more flexible channel encoding, i.e., Tx/Rx module, is crucial for CIFAR10.

5.4.3. Channel Uses Constraint

Since one of the main advantages of semantic communication lies in savings of information rate, we also investigate the influence of the number of channel uses $N_{\mathrm{Tx}}$ on MNIST classification error rate shown in Figure 6. From a practical point of view, we fix the information bottleneck by the output dimension $N_{\mathrm{Tx}}$ and maximize the mutual information $I_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)$. Decreasing the number of channel uses from $N_{\mathrm{Tx}}=14$ to 2 and accordingly the upper bound $I_{\mathrm{C}}$ on the mutual information $I_{\mathit{\theta }}\left(\mathbf{s};\mathbf{y}\right)$, i.e., compression rate, from (19) or (20), we observe that the error floor at high SNR increases. We assume that, since the channel capacity decreases with SNR and $N_{\mathrm{Tx}}$, higher compression is required for reliable transmission through the channel in the training SNR interval. For $N_{\mathrm{Tx}}=56$, almost no error floor occurs at the cost of a smaller channel encoding gain. This means compression and channel coding are balanced based on the channel condition, i.e., training SNR region, to find the optimal trade-off to maximize $I_{\mathit{\theta }}\left(\mathbf{z};\mathbf{y}\right)$, which we can also observe in unshown simulations.

5.4.4. Semantic vs. Classic Design

Finally, we compare semantic and classic communication system designs. For the classic digital design, we first assume that the images are compressed lossless and protected by a channel code for transmission and reliable overall image classification by the central unit based on $q_{\mathit{\phi }}\left(\mathbf{z}\right|\mathbf{s})$ (Central image com.). We apply Huffman encoding to a block containing 100 images ${\mathsf{s}}_i$ where each RGB color entry contains 8 bits.

For fairness, we also compare to a SINFONY version where Tx and Rx modules of Table 1 are replaced by a classic design (SINFONY - Classic digital com.). We first compress each element of the feature vector ${\mathbf{r}}_i$ that is computed in 32-bit floating-point precision in the distributed setting SINFONY - AWGN to 16-bit. Then, we apply Huffman encoding to a block containing 100 feature vectors of length $N_{\mathrm{Feat}}$.

Further, we use a 5G LDPC channel code implementation from [52] with interleaver, rate $R_{\mathrm{C}}=\{0.25,0.5,0.25\}$ and long block length of $\{15360,16000,15360\}$, and modulate the code bits with {BPSK, BPSK, 16-QAM} such that we have, e.g., parameter set $\{0.25,15360,\mathrm{BPSK}\}$ in one simulation. For digital image transmission, we use a rate of $R_{\mathrm{C}}=0.25$ with a block length of 15360 and BPSK modulation. At the receiver, we assume belief propagation decoding, where the noise variance is perfectly known for LLR computation.

The results in Figure 7 reveal tremendous information rate savings for the semantic design with SINFONY. We observe an enormous SNR shift of roughly 20 dB compared to the classic digital design with regard to to both image (Central image com.) and feature transmission (SINFONY - Classic digital com.). Note that the classic design is already near the Shannon limit and even if we improve it by ML we are only able to shift its curve by a few dB. The reason may lie in overall system optimization with SINFONY with regard to semantics and analog encoding of $\mathbf{x}$.

5.4.5. SINFONY vs. Analog “Semantic” Autoencoder

To distinguish both influences, we also implemented the approach of (14) according to Shannon by analog AEs. The analog AE has been introduced by O’Shea and Hoydis in [6]. From the viewpoint of semantic communication, it resembles the semantic approach from [19,21,28,32] without differentiating between semantic and channel coding, and the mutual information constraint $I_{}\left(\mathbf{x};\mathbf{y}\right)$ like in [21]. We trained the AE matching the Tx and Rx module in Table 1 with mean square error criterion for reliable transmission of the feature vector $\mathbf{r}$ with SINFONY training settings. The Rx module consists of one ReLU layer of width $N_{\mathrm{w}}=N_{\mathrm{Tx}}$ providing the estimate of $\mathbf{r}$. We provide results (SINFONY - Analog semantic AE) in Figure 7. Indeed, most of the shift is due to analog encoding. By this means, we further avoid the typical thresholding behavior of a classic digital system seen at 14 dB.

In conclusion, this surprisingly clear result justifies an analog “semantic” communications design and shows its huge potential to provide bandwidth savings. However, introducing the semantic RV $\mathbf{z}$ by SINFONY, we can further shift the curve by 2 dB and avoid a slightly higher error floor compared to the analog “semantic” AE. With expect a larger performance gap with more challenging image datasets, such as CIFAR10. More importantly, the main benefit of SINFONY lies in lower training complexity. We avoid separate and possibly iterative semantic and communication training procedures where in the first step we need to train SINFONY with ideal links hard to achieve in practice.

6. Conclusions

Motivated by the approach of Bao, Basu et al. [16,17] and inspired by Weaver’s notion of semantic communication [2], we brought the terminus of a semantic source to the context of communications by considering its complete Markov chain. We defined the task of semantic communication in the sense of a data-reduced and reliable transmission of communications sources/messages over a communication channel such that the semantic Random Variable (RV) at a recipient is best preserved. We formulated its design either as an Information Maximization or as an Information Bottleneck optimization problem covering important implementations aspects like the reparametrization trick and solved the problems approximately by minimizing the cross entropy that upper bounds the negative mutual information. With this article, we distinguish from related literature [16,17,21,26,32] in both classification and perspective of semantic communication and a different ML-based solution approach.

Finally, we proposed the ML-based semantic communication system SINFONY for a distributed multipoint scenario: SINFONY communicates the meaning behind multiple messages that are observed at different senders to a single receiver for semantic recovery. We analyzed SINFONY by processing images as an example of messages. Notably, numerical results reveal a tremendous rate-normalized SNR shift up to 20 dB compared to classically designed communication systems.

Outlook

In this work, we contributed to the theoretical problem description of semantic communication and data-based ML solution approaches with DNNs. There remain open research questions such as:

• Numerical Comparison to Variational IB: It remains unclear if solving the variational IB problem (21) holds benefits compared to our proposed approach.
• Implementation: Optimization with the reparametrization trick requires a known differential channel model and training at one location with dedicated hardware such as graphics processing units [53]. In addition, large amounts of labeled data are required with data-driven ML techniques, which can be expensive and time-consuming to acquire and process. Hence, further research is required to clarify how a semantic design can be implemented efficiently in practice.
• Semantic Modeling: Developing effective models of semantics is crucial, and thus we proposed the usage of probabilistic models. If the underlying problem can be described by a well-known model, e.g., a physical process to be measured and processed by a sensor network [32], a promising idea is to apply model-based approaches based on Bayesian inference for encoding and decoding—potentially combined with the technique of deep unfolding. In the context of NLP, design of knowledge graphs such as ontologies or taxonomies is a promising modeling approach for human language.
• Inconsistent Knowledge Bases: We assumed that sender and recipient share the same background knowledge base: How does performance deteriorate if there is a mismatch and how to deal with this problem [27]?