Next Article in Journal
Improving Perceived Quality of Live Adaptative Video Streaming
Previous Article in Journal
Entropy as an Objective Function of Optimization Multimodal Transportations
 
Search for Articles:
Title / Keyword
Author / Affiliation
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
Journals
Entropy
Volume 23
Issue 8
10.3390/e23080947
entropy-logo
Submit to this Journal Review for this Journal Edit a Special Issue
► Article Menu

Article Menu

share announcement question_answer thumb_up
...
textsms
...
Article

The Entropy Gain of Linear Systems and Some of Its Implications

by
Milan S. Derpich
1,*,
Matias Müller
1,* and
Jan Østergaard
2,*
1
Department of Electronic Engineering, Universidad Técnica Federico Santa María, Av. España 1680, Valparaíso 2390123, Chile
2
Department of Electronic Systems, Aalborg University, 9220 Aalborg, Denmark
*
Authors to whom correspondence should be addressed.
Academic Editor: Philip Broadbridge
Entropy 2021, 23(8), 947; https://doi.org/10.3390/e23080947
Received: 23 May 2021 / Revised: 12 July 2021 / Accepted: 20 July 2021 / Published: 24 July 2021
(This article belongs to the Section Information Theory, Probability and Statistics)
View Full-Text Download PDF
Browse Figures

Abstract

We study the increase in per-sample differential entropy rate of random sequences and processes after being passed through a non minimum-phase (NMP) discrete-time, linear time-invariant (LTI) filter G. For LTI discrete-time filters and random processes, it has long been established by Theorem 14 in Shannon’s seminal paper that this entropy gain, G(G), equals the integral of log|G(ejω)|. In this note, we first show that Shannon’s Theorem 14 does not hold in general. Then, we prove that, when comparing the input differential entropy to that of the entire (longer) output of G, the entropy gain equals G(G). We show that the entropy gain between equal-length input and output sequences is upper bounded by G(G) and arises if and only if there exists an output additive disturbance with finite differential entropy (no matter how small) or a random initial state. Unlike what happens with linear maps, the entropy gain in this case depends on the distribution of all the signals involved. We illustrate some of the consequences of these results by presenting their implications in three different problems. Specifically: conditions for equality in an information inequality of importance in networked control problems; extending to a much broader class of sources the existing results on the rate-distortion function for non-stationary Gaussian sources, and an observation on the capacity of auto-regressive Gaussian channels with feedback. View Full-Text
Keywords: differential entropy rate; non-minimum phase linear time-invariant systems; entropy loss in linear filters; networked control; rate-distortion for non-stationary sources feedback capacity differential entropy rate; non-minimum phase linear time-invariant systems; entropy loss in linear filters; networked control; rate-distortion for non-stationary sources feedback capacity
►▼ Show Figures

Figure 1

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited
SciFeed

Share and Cite

MDPI and ACS Style

Derpich, M.S.; Müller, M.; Østergaard, J. The Entropy Gain of Linear Systems and Some of Its Implications. Entropy 2021, 23, 947. https://doi.org/10.3390/e23080947

AMA Style

Derpich MS, Müller M, Østergaard J. The Entropy Gain of Linear Systems and Some of Its Implications. Entropy. 2021; 23(8):947. https://doi.org/10.3390/e23080947

Chicago/Turabian Style

Derpich, Milan S., Matias Müller, and Jan Østergaard. 2021. "The Entropy Gain of Linear Systems and Some of Its Implications" Entropy 23, no. 8: 947. https://doi.org/10.3390/e23080947

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Metrics

Article Access Map by Country/Region

1
Back to TopTop