Entropy

14 pages, 1280 KiB

Open AccessArticle

Further Exploration of an Upper Bound for Kemeny’s Constant

by Robert E. Kooij and Johan L. A. Dubbeldam

Entropy 2025, 27(4), 384; https://doi.org/10.3390/e27040384 - 4 Apr 2025

Even though Kemeny’s constant was first discovered in Markov chains and expressed by Kemeny in terms of mean first passage times on a graph, it can also be expressed using the pseudo-inverse of the Laplacian matrix representing the graph, which facilitates the calculation [...] Read more.

Even though Kemeny’s constant was first discovered in Markov chains and expressed by Kemeny in terms of mean first passage times on a graph, it can also be expressed using the pseudo-inverse of the Laplacian matrix representing the graph, which facilitates the calculation of a sharp upper bound of Kemeny’s constant. We show that for certain classes of graphs, a previously found bound is tight, which generalises previous results for bipartite and (generalised) windmill graphs. Moreover, we show numerically that for real-world networks, this bound can be used to find good numerical approximations for Kemeny’s constant. For certain graphs consisting of up to 100 K nodes, we find a speedup of a factor 30, depending on the accuracy of the approximation that can be achieved. For networks consisting of over 500 K nodes, the approximation can be used to estimate values for the Kemeny constant, where exact calculation is no longer feasible within reasonable computation time. Full article

(This article belongs to the Special Issue Complexity, Entropy and the Physics of Information II)

20 pages, 525 KiB

Open AccessArticle

Stochastic Entropy Production for Classical and Quantum Dynamical Systems with Restricted Diffusion

by Jonathan Dexter and Ian J. Ford

Entropy 2025, 27(4), 383; https://doi.org/10.3390/e27040383 - 3 Apr 2025

Abstract

Modeling the evolution of a system using stochastic dynamics typically implies increasing subjective uncertainty in the adopted state of the system and its environment as time progresses, and stochastic entropy production has been developed as a measure of this change. In some situations, [...] Read more.

Modeling the evolution of a system using stochastic dynamics typically implies increasing subjective uncertainty in the adopted state of the system and its environment as time progresses, and stochastic entropy production has been developed as a measure of this change. In some situations, the evolution of stochastic entropy production can be described using an Itô process, but mathematical difficulties can emerge if diffusion in the system phase space happens to be restricted to a subspace of a lower dimension. This situation can arise if there are constants of the motion, for example, or more generally when there are functions of the coordinates that evolve without noise. More simply, difficulties can emerge if there are more coordinates than there are independent noises. We show how the problem of computing the stochastic entropy production in such a situation can be overcome. We illustrate the approach using a simple case of diffusion on an ellipse. We go on to consider an open three-level quantum system modeled within a framework of Markovian quantum state diffusion. We show how a nonequilibrium stationary state of the system, with a constant mean rate of stochastic entropy production, can be established under suitable environmental couplings. Full article

(This article belongs to the Special Issue Entropy: From Atoms to Complex Systems)

18 pages, 8833 KiB

Open AccessArticle

Finding Key Nodes in Complex Networks Through Quantum Deep Reinforcement Learning

by Juechan Xiong, Xiao-Long Ren and Linyuan Lü

Entropy 2025, 27(4), 382; https://doi.org/10.3390/e27040382 - 3 Apr 2025

Abstract

Identifying key nodes in networks is a fundamental problem in network science. This study proposes a quantum deep reinforcement learning (QDRL) framework that integrates reinforcement learning with a variational quantum graph neural network, effectively identifying distributed influential nodes while preserving the network’s fundamental [...] Read more.

Identifying key nodes in networks is a fundamental problem in network science. This study proposes a quantum deep reinforcement learning (QDRL) framework that integrates reinforcement learning with a variational quantum graph neural network, effectively identifying distributed influential nodes while preserving the network’s fundamental topological properties. By leveraging principles of quantum computing, our method is designed to reduce model parameters and computational complexity compared to traditional neural networks. Trained on small networks, it demonstrated strong generalization across diverse scenarios. We compared the proposed algorithm with some classical node ranking and network dismantling algorithms on various synthetical and empirical networks. The results suggest that the proposed algorithm outperforms existing baseline methods. Moreover, in synthetic networks based on Erdős–Rényi and Watts–Strogatz models, QDRL demonstrated its capability to alleviate the issue of localization in network information propagation and node influence ranking. Our research provides insights into addressing fundamental problems in complex networks using quantum machine learning, demonstrating the potential of quantum approaches for network analysis tasks. Full article

(This article belongs to the Topic Computational Complex Networks)

► Show Figures

Figure 1

31 pages, 12545 KiB

Open AccessArticle

Complexity Analysis of Environmental Time Series

by Holger Lange and Michael Hauhs

Entropy 2025, 27(4), 381; https://doi.org/10.3390/e27040381 - 3 Apr 2025

Abstract

Small, forested catchments are prototypes of terrestrial ecosystems and have been studied in several disciplines of environmental science over several decades. Time series of water and matter fluxes and nutrient concentrations from these systems exhibit a bewildering diversity of spatiotemporal patterns, indicating the [...] Read more.

Small, forested catchments are prototypes of terrestrial ecosystems and have been studied in several disciplines of environmental science over several decades. Time series of water and matter fluxes and nutrient concentrations from these systems exhibit a bewildering diversity of spatiotemporal patterns, indicating the intricate nature of processes acting on a large range of time scales. Nonlinear dynamics is an obvious framework to investigate catchment time series. We analyzed selected long-term data from three headwater catchments in the Bramke valley, Harz mountains, Lower Saxony in Germany at common biweekly resolution for the period 1991 to 2023. For every time series, we performed gap filling, detrending, and removal of the annual cycle using singular system analysis (SSA), and then calculated metrics based on ordinal pattern statistics: the permutation entropy, permutation complexity, and Fisher information, as well as their generalized versions (q-entropy and α-entropy). Further, the position of each variable in Tarnopolski diagrams is displayed and compared to reference stochastic processes, like fractional Brownian motion, fractional Gaussian noise, and β noise. Still another way of distinguishing deterministic chaos and structured noise, and quantifying the latter, is provided by the complexity from ordinal pattern positioned slopes (COPPS). We also constructed horizontal visibility graphs and estimated the exponent of the decay of the degree distribution. Taken together, the analyses create a characterization of the dynamics of these systems which can be scrutinized for universality, either across variables or between the three geographically very close catchments. Full article

(This article belongs to the Special Issue Nonlinear Dynamics and Applications—In Honor of Professor Osvaldo Anibal Rosso's 70th Birthday)

► Show Figures

Figure 1

16 pages, 1082 KiB

Open AccessArticle

Adaptive Kalman Filtering Localization Calibration Method Based on Dynamic Mutation Perception and Collaborative Correction

by Zijia Huang, Qiushi Xu, Menghao Sun, Xuzhen Zhu and Shaoshuai Fan

Entropy 2025, 27(4), 380; https://doi.org/10.3390/e27040380 - 3 Apr 2025

Abstract

Aiming at the problem of reduced positioning accuracy of unmanned swarm navigation systems due to dynamic abrupt noise in a complex electromagnetic environment, this paper proposes an adaptive Kalman filtering positioning and calibration method based on dynamic mutation perception and collaborative correction. This [...] Read more.

Aiming at the problem of reduced positioning accuracy of unmanned swarm navigation systems due to dynamic abrupt noise in a complex electromagnetic environment, this paper proposes an adaptive Kalman filtering positioning and calibration method based on dynamic mutation perception and collaborative correction. This method optimizes the performance of Kalman filtering by monitoring the mutation of acceleration and velocity in real time, designing a dynamic threshold detection mechanism, adaptively adjusting the covariance matrix, and using multidimensional scaling analysis to calculate the similarity of trajectories and collaboratively correct the current state. The experiment uses simulation and real scene data and compares algorithms such as the traditional extended Kalman filter to verify the effectiveness of the proposed method, providing an effective solution for the collaborative positioning of an unmanned swarm under complex electromagnetic interference. Full article

(This article belongs to the Special Issue Complexity, Entropy and the Physics of Information II)

► Show Figures

Figure 1

16 pages, 2258 KiB

Open AccessArticle

Lightweight Pre-Trained Korean Language Model Based on Knowledge Distillation and Low-Rank Factorization

by Jin-Hwan Kim and Young-Seok Choi

Entropy 2025, 27(4), 379; https://doi.org/10.3390/e27040379 - 2 Apr 2025

Abstract

Natural Language Processing (NLP) stands as a forefront of artificial intelligence research, empowering computational systems to comprehend and process human language as used in everyday contexts. Language models (LMs) underpin this field, striving to capture the intricacies of linguistic structure and semantics by [...] Read more.

Natural Language Processing (NLP) stands as a forefront of artificial intelligence research, empowering computational systems to comprehend and process human language as used in everyday contexts. Language models (LMs) underpin this field, striving to capture the intricacies of linguistic structure and semantics by assigning probabilities to sequences of words. The trend towards large language models (LLMs) has shown significant performance improvements with increasing model size. However, the deployment of LLMs on resource-limited devices such as mobile and edge devices remains a challenge. This issue is particularly pronounced in languages other than English, including Korean, where pre-trained models are relatively scarce. Addressing this gap, we introduce a novel lightweight pre-trained Korean language model that leverages knowledge distillation and low-rank factorization techniques. Our approach distills knowledge from a 432 MB (approximately 110 M parameters) teacher model into student models of substantially reduced sizes (e.g., 53 MB ≈ 14 M parameters, 35 MB ≈ 13 M parameters, 30 MB ≈ 11 M parameters, and 18 MB ≈ 4 M parameters). The smaller student models further employ low-rank factorization to minimize the parameter count within the Transformer’s feed-forward network (FFN) and embedding layer. We evaluate the efficacy of our lightweight model across six established Korean NLP tasks. Notably, our most compact model, KR-ELECTRA-Small-KD, attains over 97.387% of the teacher model’s performance despite an 8.15× reduction in size. Remarkably, on the NSMC sentiment classification benchmark, KR-ELECTRA-Small-KD surpasses the teacher model with an accuracy of 89.720%. These findings underscore the potential of our model as an efficient solution for NLP applications in resource-constrained settings. Full article

(This article belongs to the Special Issue Information Processing in Complex Biological Systems)

► Show Figures

Figure 1

47 pages, 2484 KiB

Open AccessArticle

Exploring ISAC: Information-Theoretic Insights

by Mehrasa Ahmadipour, Michèle Wigger and Shlomo Shamai

Entropy 2025, 27(4), 378; https://doi.org/10.3390/e27040378 - 2 Apr 2025

Abstract

This article reviews results from the literature illustrating the bottlenecks and tradeoffs of integrated sensing and communication (ISAC) through the lens of information theory, thus offering a distinct perspective compared to recent works that focus on signal processing, wireless communications, or other related [...] Read more.

This article reviews results from the literature illustrating the bottlenecks and tradeoffs of integrated sensing and communication (ISAC) through the lens of information theory, thus offering a distinct perspective compared to recent works that focus on signal processing, wireless communications, or other related overviews. Different models and scenarios are considered and compared. For example, scenarios where radar sensing is performed at the communication and radar transmitter (mono-static ISAC) and scenarios where the radar receiver differs from the radar transmitter (called bi-static radar). Similarly, we discuss ISAC bottlenecks and tradeoffs both in slowly-varying environments where the main sensing target is described by a single parameter and accordingly, sensing performance is described by detection error probabilities, as well as in fast-varying environments, where the sensing targets are described by vectors and thus vector-valued performance measures such as average distortions like mean-squared errors are used to determine sensing performances. This overview article further also considers limitations and opportunities in network ISAC environments, such as collaborative or interactive sensing, and the influence of secrecy and privacy requirements on ISAC systems, a line of research that has received growing interest over the last few years. For all these scenarios, we provide and discuss precise models and their limitations and provide either bounds or full characterizations of the fundamental information-theoretic performance limits of these systems. Further extensions as well as important open research directions are also discussed. Full article

(This article belongs to the Special Issue Integrated Sensing and Communications)

21 pages, 1375 KiB

Open AccessArticle

Bit-Level Construction for Multiplicative-Repetition-Based Non-Binary Polar Codes

by Rongchi Xu, Peiyao Chen, Ling Liu, Min Zhu and Baoming Bai

Entropy 2025, 27(4), 377; https://doi.org/10.3390/e27040377 - 2 Apr 2025

Abstract

In this paper, we discuss non-binary polar codes using a

2 \times 2

matrix over a Galois field GF(

2^{q}

) as the kernel. Conventional construction of non-binary polar codes divides the synthesized channels into frozen channels and information channels. Each information [...] Read more.

In this paper, we discuss non-binary polar codes using a

2 \times 2

matrix over a Galois field GF(

2^{q}

) as the kernel. Conventional construction of non-binary polar codes divides the synthesized channels into frozen channels and information channels. Each information channel carries one symbol, i.e., q bits. However, there are many middle channels with insufficient polarization, which cannot carry one symbol of q bits but only i bits,

1 \leq i < q, i \in Z

, at finite block length. In this paper, we consider bit-level construction for multiplicative repetition (MR)-based non-binary polar codes and propose a bit-level construction based on the two following methods. We first calculate the error probability and channel capacity lower bound of each synthesized channel based on the channel degradation method, and then determine both the number and index of the carried bits for each synthesized channel according to the symbol error probability and capacity. To reduce complexity, we also introduce a Monte-Carlo method. We compute the error probability of each synthesized channel carrying i information bits and select the optimal construction that can minimize the union bound of the error probability. Finally, an improved construction-based probabilistic shaping method for MR-based non-binary polar codes is considered. Simulation results show that the proposed construction significantly improved the decoding performance compared with the conventional construction scheme. Full article

(This article belongs to the Special Issue Advances in Modern Channel Coding)

► Show Figures

Figure 1

26 pages, 464 KiB

Open AccessArticle

A Family of Optimal Linear Functional-Repair Regenerating Storage Codes

by Henk D. L. Hollmann

Entropy 2025, 27(4), 376; https://doi.org/10.3390/e27040376 - 1 Apr 2025

Abstract

We construct a family of linear optimal functional-repair regenerating storage codes with parameters [...] Read more.

We construct a family of linear optimal functional-repair regenerating storage codes with parameters

{m, (n, k), (r, α, β)} = {(2 r - α + 1) α / 2, (r + 1, r), (r, α, 1)}

for any integers (

r, α

) with

1 \leq α \leq r

over any field when

α \in {1, r - 1, r}

and over any finite field (

F_{q}

) with

q \geq r - 1

otherwise. These storage codes are Minimum-Storage Regenerating (MSR) when

α = 1

and Minimum-Bandwidth Regenerating (MBR) when

α = r

and represent extremal points of the (convex) attainable cut-set region different from the MSR and MBR points in all other cases. It is known that when

2 \leq α \leq r - 1

, these parameters cannot be realized by exact-repair storage codes. Each of these codes comes with an explicit and relatively simple repair method, and repair can even be realized as help by transfer (HBT) if desired. The coding states of codes from this family can be described geometrically as configurations of

r + 1

subspaces of dimension

α

in an m-dimensional vector space with restricted sub-span dimensions. A few “small” codes with these parameters are known: one for

(r, α) = (3, 2)

dating from 2013 and one for

(r, α) = (4, 3)

dating from 2024. Apart from these, our codes are the first examples of explicit, relatively simple, optimal functional-repair storage codes over a small, finite field with an explicit repair method and with parameters representing an extremal point of the attainable cut-set region distinct from the MSR and MBR points. Full article

(This article belongs to the Special Issue Discrete Math in Coding Theory)

► Show Figures

Figure 1

20 pages, 3249 KiB

Open AccessArticle

Correlations and Kappa Distributions: Numerical Experiment and Physical Understanding

by David J. McComas, George Livadiotis and Nicholas V. Sarlis

Entropy 2025, 27(4), 375; https://doi.org/10.3390/e27040375 - 31 Mar 2025

Abstract

Kappa distributions, their statistical framework, and their thermodynamic origin describe systems with correlations among their particle energies, residing in stationary states out of classical thermal equilibrium/space plasmas, from solar wind to the outer heliosphere, are such systems. We show how correlations from long-range [...] Read more.

Kappa distributions, their statistical framework, and their thermodynamic origin describe systems with correlations among their particle energies, residing in stationary states out of classical thermal equilibrium/space plasmas, from solar wind to the outer heliosphere, are such systems. We show how correlations from long-range interactions compete with collisions to define the specific shape of particle velocity distributions, using a simple numerical experiment with collisions and a variable amount of correlation among the particles. When the correlations are turned off, collisions drive any initial distribution to evolve toward equilibrium and a Maxwell–Boltzmann (MB) distribution. However, when some correlation is introduced, the distribution evolves toward a different stationary state defined by a kappa distribution with some finite value of the thermodynamic kappa

κ

(where

κ \to \infty

corresponds to a MB distribution). Furthermore, the stronger the correlations, the lower the

κ

value. This simple numerical experiment illuminates the role of correlations in forming stationary state particle distributions, which are described by kappa distributions, as well as the physical interpretation of correlations from long-range interactions and how they are related to the thermodynamic kappa. Full article

(This article belongs to the Collection Foundations of Statistical Mechanics)

► Show Figures

Figure 1

12 pages, 10013 KiB

Open AccessArticle

Transient Thermal Energy Harvesting at a Single Temperature Using Nonlinearity

by Tamzeed B. Amin, James M. Mangum, Md R. Kabir, Syed M. Rahman, Ashaduzzaman, Pradeep Kumar, Luis L. Bonilla and Paul M. Thibado

Entropy 2025, 27(4), 374; https://doi.org/10.3390/e27040374 - 31 Mar 2025

Abstract

The authors present an in-depth theoretical study of two nonlinear circuits capable of transient thermal energy harvesting at one temperature. The first circuit has a storage capacitor and diode connected in series. The second circuit has three storage capacitors, and two diodes arranged [...] Read more.

The authors present an in-depth theoretical study of two nonlinear circuits capable of transient thermal energy harvesting at one temperature. The first circuit has a storage capacitor and diode connected in series. The second circuit has three storage capacitors, and two diodes arranged for full wave rectification. The authors solve both Ito–Langevin and Fokker–Planck equations for both circuits using a large parameter space including capacitance values and diode quality. Surprisingly, using diodes one can harvest thermal energy at a single temperature by charging capacitors. However, this is a transient phenomenon. In equilibrium, the capacitor charge is zero, and this solution alone satisfies the second law of thermodynamics. The authors found that higher quality diodes provide more stored charge and longer lifetimes. Harvesting thermal energy from the ambient environment using diode nonlinearity requires capacitors to be charged but then disconnected from the circuit before they have time to discharge. Full article

(This article belongs to the Section Thermodynamics)

► Show Figures

Figure 1

30 pages, 1467 KiB

Open AccessArticle

Rate–Distortion–Perception Trade-Off in Information Theory, Generative Models, and Intelligent Communications

by Xueyan Niu, Bo Bai, Nian Guo, Weixi Zhang and Wei Han

Entropy 2025, 27(4), 373; https://doi.org/10.3390/e27040373 - 31 Mar 2025

Abstract

Traditional rate–distortion (RD) theory examines the trade-off between the average length of the compressed representation of a source and the additive distortions of its reconstruction. The rate–distortion–perception (RDP) framework, which integrates the perceptual dimension into the RD paradigm, has garnered significant attention due [...] Read more.

Traditional rate–distortion (RD) theory examines the trade-off between the average length of the compressed representation of a source and the additive distortions of its reconstruction. The rate–distortion–perception (RDP) framework, which integrates the perceptual dimension into the RD paradigm, has garnered significant attention due to recent advancements in machine learning, where perceptual fidelity is assessed by the divergence between input and reconstruction distributions. In communication systems where downstream tasks involve generative modeling, high perceptual fidelity is essential, despite distortion constraints. However, while zero distortion implies perfect realism, the converse is not true, highlighting an imbalance in the significance of distortion and perceptual constraints. This article clarifies that incorporating perceptual constraints does not decrease the necessary rate; instead, under certain conditions, additional rate is required, even with the aid of common and private randomness, which are key elements in generative models. Consequently, we project an increase in expected traffic in intelligent communication networks with the consideration of perceptual quality. Nevertheless, a modest increase in rate can enable generative models to significantly enhance the perceptual quality of reconstructions. By exploring the synergies between generative modeling and communication through the lens of information-theoretic results, this article demonstrates the benefits of intelligent communication systems and advocates for the application of the RDP framework in advancing compression and semantic communication research. Full article

(This article belongs to the Special Issue Semantic Information Theory)

► Show Figures

Figure 1

14 pages, 267 KiB

Open AccessArticle

Intrinsic Motivation as Constrained Entropy Maximization

by Alex B. Kiefer

Entropy 2025, 27(4), 372; https://doi.org/10.3390/e27040372 - 31 Mar 2025

Abstract

“Intrinsic motivation” refers to the capacity for intelligent systems to be motivated endogenously, i.e., by features of agential architecture itself rather than by learned associations between action and reward. This paper views active inference, empowerment, and other formal accounts of intrinsic motivation as [...] Read more.

“Intrinsic motivation” refers to the capacity for intelligent systems to be motivated endogenously, i.e., by features of agential architecture itself rather than by learned associations between action and reward. This paper views active inference, empowerment, and other formal accounts of intrinsic motivation as variations on the theme of constrained maximum entropy inference, providing a general perspective on intrinsic motivation complementary to existing frameworks. The connection between free energy and empowerment noted in previous literature is further explored, and it is argued that the maximum-occupancy approach in practice incorporates an implicit model-evidence constraint. Full article

(This article belongs to the Special Issue From Functional Imaging to Free Energy—Dedicated to Professor Karl Friston on the Occasion of His 65th Birthday)

29 pages, 1945 KiB

Open AccessArticle

Latent Abstractions in Generative Diffusion Models

by Giulio Franzese, Mattia Martini, Giulio Corallo, Paolo Papotti and Pietro Michiardi

Entropy 2025, 27(4), 371; https://doi.org/10.3390/e27040371 - 31 Mar 2025

Abstract

In this work, we study how diffusion-based generative models produce high-dimensional data, such as images, by relying on latent abstractions that guide the generative process. We introduce a novel theoretical framework extending Nonlinear Filtering (NLF), offering a new perspective on SDE-based generative models. [...] Read more.

In this work, we study how diffusion-based generative models produce high-dimensional data, such as images, by relying on latent abstractions that guide the generative process. We introduce a novel theoretical framework extending Nonlinear Filtering (NLF), offering a new perspective on SDE-based generative models. Our theory is based on a new formulation of joint (state and measurement) dynamics and an information-theoretic measure of state influence on the measurement process. We show that diffusion models can be interpreted as a system of SDE, describing a non-linear filter where unobservable latent abstractions steer the dynamics of an observable measurement process. Additionally, we present an empirical study validating our theory and supporting previous findings on the emergence of latent abstractions at different generative stages. Full article

(This article belongs to the Special Issue The Statistical Physics of Generative Diffusion Models)

► Show Figures

Figure 1

14 pages, 265 KiB

Open AccessArticle

Successive Refinement for Lossy Compression of Individual Sequences

by Neri Merhav

Entropy 2025, 27(4), 370; https://doi.org/10.3390/e27040370 - 31 Mar 2025

Abstract

We consider the problem of successive-refinement coding for lossy compression of individual sequences, namely, compression in two stages, where in the first stage, a coarse description at a relatively low rate is sent from the encoder to the decoder, and in the second [...] Read more.

We consider the problem of successive-refinement coding for lossy compression of individual sequences, namely, compression in two stages, where in the first stage, a coarse description at a relatively low rate is sent from the encoder to the decoder, and in the second stage, an additional coding rate is allocated in order to refine the description and thereby improve the reproduction. Our main result is in establishing outer bounds (converse theorems) for the rate region where we limit the encoders to be finite-state machines in the spirit of Ziv and Lempel’s 1978 model. The matching achievability scheme is conceptually straightforward. We also consider the more general multiple description coding problem on a similar footing and propose achievability schemes that are analogous to the well-known El Gamal–Cover and the Zhang–Berger achievability schemes of memoryless sources and additive distortion measures. Full article

(This article belongs to the Collection Feature Papers in Information Theory)

23 pages, 1113 KiB

Open AccessArticle

Feature-Driven Semantic Communication for Efficient Image Transmission

by Ji Zhang, Ying Zhang, Baofeng Ji, Anmin Chen, Aoxue Liu and Hengzhou Xu

Entropy 2025, 27(4), 369; https://doi.org/10.3390/e27040369 - 31 Mar 2025

Abstract

Semantic communication is an emerging approach that enhances transmission efficiency by conveying the semantic content of information more effectively. It has garnered significant attention in recent years. However, existing semantic communication systems for image transmission typically adopt direct transmission of features or uniformly [...] Read more.

Semantic communication is an emerging approach that enhances transmission efficiency by conveying the semantic content of information more effectively. It has garnered significant attention in recent years. However, existing semantic communication systems for image transmission typically adopt direct transmission of features or uniformly compress features before transmission. They have not yet considered the differential impact of features on image recovery at the receiver end and the issue of bandwidth limitations during actual transmission. This paper shows that non-uniform processing of features leads to better image recovery under bandwidth constraints compared to uniform processing. Based on this, we propose a semantic communication system for image transmission, which introduces non-uniform quantization techniques. In the feature transmission stage, the system performs varying levels of quantization based on the differences in feature performance at the receiver, thereby reducing the bandwidth requirement. Inspired by quantitative quantization techniques, we design a non-uniform quantization algorithm capable of dynamic bit allocation. This algorithm, under bandwidth constraints, dynamically adjusts the quantization precision of features based on their contribution to the completion of tasks at the receiver end, ensuring the quality and accuracy of the transmitted data even under limited bandwidth conditions. Experimental results show that the proposed system reduces bandwidth usage while ensuring image reconstruction quality. Full article

(This article belongs to the Special Issue Semantic Information Theory)

► Show Figures

Figure 1

19 pages, 6626 KiB

Open AccessArticle

Action Recognition with 3D Residual Attention and Cross Entropy

by Yuhao Ouyang and Xiangqian Li

Entropy 2025, 27(4), 368; https://doi.org/10.3390/e27040368 - 31 Mar 2025

Abstract

This study proposes a three-dimensional (3D) residual attention network (3DRFNet) for human activity recognition by learning spatiotemporal representations from motion pictures. Core innovation integrates the attention mechanism into the 3D ResNet framework to emphasize key features and suppress irrelevant ones. In each 3D [...] Read more.

This study proposes a three-dimensional (3D) residual attention network (3DRFNet) for human activity recognition by learning spatiotemporal representations from motion pictures. Core innovation integrates the attention mechanism into the 3D ResNet framework to emphasize key features and suppress irrelevant ones. In each 3D ResNet block, channel and spatial attention mechanisms generate attention maps for tensor segments, which are then multiplied by the input feature mapping to emphasize key features. Additionally, the integration of Fast Fourier Convolution (FFC) enhances the network’s capability to effectively capture temporal and spatial features. Simultaneously, we used the cross-entropy loss function to describe the difference between the predicted value and GT to guide the model’s backpropagation. Subsequent experimental results have demonstrated that 3DRFNet achieved SOTA performance in human action recognition. 3DRFNet achieved accuracies of 91.7% and 98.7% on the HMDB-51 and UCF-101 datasets, respectively, which highlighted 3DRFNet’s advantages in recognition accuracy and robustness, particularly in effectively capturing key behavioral features in videos using both attention mechanisms. Full article

(This article belongs to the Special Issue Application of Information Theory to Computer Vision and Image Processing II)

► Show Figures

Figure 1

23 pages, 8280 KiB

Open AccessArticle

From Task Distributions to Expected Paths Lengths Distributions: Value Function Initialization in Sparse Reward Environments for Lifelong Reinforcement Learning

by Soumia Mehimeh and Xianglong Tang

Entropy 2025, 27(4), 367; https://doi.org/10.3390/e27040367 - 30 Mar 2025

Abstract

This paper studies value function transfer within reinforcement learning frameworks, focusing on tasks continuously assigned to an agent through a probabilistic distribution. Specifically, we focus on environments characterized by sparse rewards with a terminal goal. Initially, we propose and theoretically demonstrate that the [...] Read more.

This paper studies value function transfer within reinforcement learning frameworks, focusing on tasks continuously assigned to an agent through a probabilistic distribution. Specifically, we focus on environments characterized by sparse rewards with a terminal goal. Initially, we propose and theoretically demonstrate that the distribution of the computed value function from such environments, whether in cases where the goals or the dynamics are changing across tasks, can be reformulated as the distribution of the number of steps to the goal generated by their optimal policies, which we name the expected optimal path length. To test our propositions, we hypothesize that the distribution of the expected optimal path lengths resulting from the task distribution is normal. This claim leads us to propose that if the distribution is normal, then the distribution of the value function follows a log-normal pattern. Leveraging this insight, we introduce “LogQInit” as a novel value function transfer method, based on the properties of log-normality. Finally, we run experiments on a scenario of goals and dynamics distributions, validate our proposition by providing an a dequate analysis of the results, and demonstrate that LogQInit outperforms existing methods of value function initialization, policy transfer, and reward shaping. Full article

(This article belongs to the Section Information Theory, Probability and Statistics)

► Show Figures

Figure 1

20 pages, 342 KiB

Open AccessArticle

Generalized Orthogonal de Bruijn and Kautz Sequences

by Yuan-Pon Chen, Jin Sima and Olgica Milenkovic

Entropy 2025, 27(4), 366; https://doi.org/10.3390/e27040366 - 30 Mar 2025

Abstract

A de Bruijn sequence of order k over a finite alphabet is a cyclic sequence with the property that it contains every possible k-sequence as a substring exactly once. Orthogonal de Bruijn sequences are the collections of de Bruijn sequences of the [...] Read more.

A de Bruijn sequence of order k over a finite alphabet is a cyclic sequence with the property that it contains every possible k-sequence as a substring exactly once. Orthogonal de Bruijn sequences are the collections of de Bruijn sequences of the same order, k, that satisfy the joint constraint that every

(k + 1)

-sequence appears as a substring in, at most, one of the sequences in the collection. Both de Bruijn and orthogonal de Bruijn sequences have found numerous applications in synthetic biology, although the latter remain largely unexplored in the coding theory literature. Here, we study three relevant practical generalizations of orthogonal de Bruijn sequences, where we relax either the constraint that every

(k + 1)

-sequence appears exactly once or the sequences themselves are de Bruijn rather than balanced de Bruijn sequences. We also provide lower and upper bounds on the number of fixed-weight orthogonal de Bruijn sequences. The paper concludes with parallel results for orthogonal nonbinary Kautz sequences, which satisfy similar constraints as de Bruijn sequences, except for being only required to cover all subsequences of length k whose maximum run length equals one. Full article

(This article belongs to the Special Issue Coding and Algorithms for DNA-Based Data Storage Systems)

► Show Figures

Figure 1

Journal Description

Entropy

Latest Articles

Journal Menu

Journal Browser

Highly Accessed Articles

Latest Books

E-Mail Alert

News

Topics

Conferences

Special Issues

Topical Collections

Further Information

Guidelines

MDPI Initiatives

Follow MDPI