Comparison of Information Criteria for Detection of Useful Signals in Noisy Environments

Abstract

This paper considers the appearance of indications of useful acoustic signals in the signal/noise mixture. Various information characteristics (information entropy, Jensen–Shannon divergence, spectral information divergence and statistical complexity) are investigated in the context of solving this problem. Both time and frequency domains are studied for the calculation of information entropy. The effectiveness of statistical complexity is shown in comparison with other information metrics for different signal-to-noise ratios. Two different approaches for statistical complexity calculations are also compared. In addition, analytical formulas for complexity and disequilibrium are obtained using entropy variation in the case of signal spectral distribution. The connection between the statistical complexity criterion and the Neyman–Pearson approach for hypothesis testing is discussed. The effectiveness of the proposed approach is shown for different types of acoustic signals and noise models, including colored noises, and different signal-to-noise ratios, especially when the estimation of additional noise characteristics is impossible.

1. Introduction

Since Shannon [1] introduced information and information entropy, these concepts have attracted significant attention from scientists, as evidenced by the large number of articles devoted to the development of information theory in relation to various theoretical and practical aspects. Many different information criteria, metrics, and methods for their calculation that are based, one way or another, on the concepts of Shannon entropy have been proposed and investigated [2]. These metrics can be used quite successfully in signal processing, which eventually led to the emergence of a separate section of this scientific area, called entropic signal analysis [3].

For signals described by time series, the information entropy can be calculated on the basis of both signal representation in the time domain [4] and its representation in the frequency domain [5], i.e., using the signal spectrum. The convenience of the second approach comes from the fact that white noise, which is usually used to model background noise in these problem statements, has a uniform frequency distribution. This allows us to simplify its mathematical description and separate useful signals more effectively.

Decision theory considers change-point detection problems, which are closely related to the problems discussed above: often, in such problems, the moment of change in the parameters of a random process registered in discrete time must be determined. In [6,7], many probabilistic-statistical methods of solving such problems are considered. The Neyman–Pearson approach to this problem was used in [8]. Additionally, one cannot ignore the so-called Anomaly Detection Problems, where detection of anomalies in time series is required [9,10,11], i.e., the moment at which the behavior of the system begins to qualitatively differ from normal for various reasons, in particular, due to unwanted external interference. The electrocardiogram (ECG) is one example of such a time series, and ECGs have been analyzed in a large number of articles, for example, [12]. The presence of an anomaly in this case can indicate health problems, and detection at an early stage may save the life of the patient.

Of particular interest is the processing of acoustic signals, which can be useful, for example, in Voice Activity Detection (VAD) problems [13] related to voice assistants. The task is usually to separate speech segments from background environmental noise. Related articles [5,14,15] present a method for endpoint detection, i.e., the determination of the limits of a speech signal in a mixture of this signal and background noise based on the calculation of the spectral entropy. The general idea of methods based on information criteria is that their values experience a sharp jump when a useful signal appears in the noise.

In a series of articles [16,17,18,19,20,21], researchers introduced the concept of a statistical measure of signal complexity, which they called statistical complexity. In [22,23], statistical complexity and information entropy were used to classify various underwater objects of animate and inanimate nature from recorded sound. In the present article, we use this measure to indicate the appearance of an useful acoustic signal in a highly noisy mixture. It should be noted that the positive side of the proposed method is that it does not require any a priori knowledge about the signal to be detected. However, a priori information, such as the approximate frequency range of the signal is known, its detection will be even more accurate.

The structure of the paper is as follows. Section 2 provides a brief theoretical summary of the information criteria used in various known signal detection methods. In Section 3, entropy variation is investigated and statistical complexity is introduced. In Section 4, the connection between statistical complexity and the Neyman–Pearson criterion for hypothesis testing is also discussed to justify the proposed approach. Section 5 is provides a variety of examples, gives a comparison of different information criteria, and discussed the results, which allows us to make an educated choice about a suitable rule for the detection of signals in a noisy mixture. Section 6 summarizes the conducted research and provides the direction for future work.

2. Information Criteria

2.1. Information Entropy and Other Information Criteria

In information theory, the entropy of a random variable is the average level of ”surprise” or ”uncertainty” inherent in the possible outcomes of the variable. For a discrete random variable X that takes values in the alphabet $\mathcal{X}$ and has a distribution density of p$p:\mathcal{X}\to [0,1]$, the entropy, according to Shannon [1], is defined as

$${\displaystyle \mathrm{H}\left(p\right)=-\sum _{x\in \mathcal{X}}p\left(x\right){log}_{2}p\left(x\right)},$$

(1)

where $\sum$ denotes the sum across all possible values of the variable. When computing the sum (1), it is agreed that $0log0=0$, and this assumption holds for all future equations. From the Formula (1), it follows that entropy reaches its maximum value when all states of the system are equally probable.

There are several definitions of information divergences, i.e., the statistical distances between two distributions. The Kullback–Leibler divergence (or mutual entropy) between two discrete probability distributions $p\left(x\right)$ and $q\left(x\right)$ on an event set $\mathcal{X}$ is defined as

$$D_{KL}\left(pq\right)=\sum _{x\in \mathcal{X}}p\left(x\right){log}_2\frac{p\left(x\right)}{q\left(x\right)}.$$

(2)

This measure is a statistical distance and distinguishes statistical processes by indicating how much $p\left(x\right)$ differs from $q\left(x\right)$ by the maximum likelihood hypothesis test when the actual data obey the distribution $p\left(x\right)$. It is easy to see that

$$D_{KL}\left(pq\right)=H(p,q)-H\left(p\right),$$

(3)

where $H(p,q)$ is a cross-entropy between p and q:

$${\displaystyle H(p,q)={\mathbb{E}}_p[-{log}_{2}q]},$$

(4)

where ${\displaystyle {\mathbb{E}}_p[·]}$ is an operator of the mathematical expectation relative to the distribution p.

The symmetrized Kullback–Leibler distance [13] is often used in studies:

$$\rho \left(pq\right)=D_{KL}\left(pq\right)+D_{KL}\left(qp\right).$$

(5)

However, the Jensen–Shannon divergence, which symmetrizes the Kullback–Leibler divergence and is often a more convenient information measure for practical applications, is used more often:

$$JSD\left(pq\right)=\frac{D_{KL}\left(pm\right)+D_{KL}\left(qm\right)}{2},m=\frac{p+q}{2}.$$

(6)

It is symmetric and always has a finite value. The square root of the Jensen–Shannon divergence is a metric that is often called the Jensen–Shannon distance.

It is easy to see that

$${\displaystyle JSD\left(pq\right)=H\left(m\right)-\frac{1}{2}(H\left(p\right)+H\left(q\right))}.$$

(7)

Another quantity related to the complexity of the system is the ”disequilibrium,” denoted by D, which shows the deviation of a given probability distribution from a uniform one. The concept of the statistical complexity of a system can be considered a development of the concept of entropy. In [16,17,18,19,20,21], it is defined as

$$C=H·D,$$

(8)

where C is the statistical complexity, H is the information entropy, and D is a measure of the disequilibrium of the distribution relative to the uniform one.

The measure of statistical complexity reflects the relationship between the amount of information and its disequilibrium in the system. As a parameter D, according to the authors of [16], one can choose any metric that determines the difference between the maximum entropy and the entropy of the studied signal. The simplest example of disequilibrium is the square of the Euclidean distance in ${\mathbb{R}}^N$ between the original distribution and the uniform distribution, but often, the Jensen–Shannon divergence [22,23] is also used.

2.2. Time Entropy

Now let us consider the information characteristics mentioned above in relation to time series. The Shannon entropy for systems with unequal probability states is defined as follows: Let the i-th state of the system have a probability of $p_i=N_i/N$, where N is a sample volume and $N_i$ is the amount of filling at the i-level. Then, the entropy $H\left(p\right)$, according to the Formula (1), equals

$$H\left(p\right)=-\sum _{i=1}^N{p}_i{log}_2{p}_i.$$

(9)

From here, we consider discrete probability distributions $p_i$ with the following properties:

$$p_i\in [0,1],\sum _{i=1}^N{p}_i=1.$$

(10)

There are different ways to calculate probabilities $p_i$ from the time series. The simplest one is as follows: First, the maximum $x_{max}$ and minimum $x_{min}$ values are found for the considered time series $x\left(t\right)$ with N data points. Then, the interval ($x_{max}-x_{min}$) is divided into n subintervals (levels) so that the value of the interval $\Delta x$ is not less than the confidence interval of the observations. The resulting sample is treated as a “message”, and the i subintervals are treated as an “alphabet”. Then, we find the number $\Delta N_i$ of sample values $x_k$ that fall into each of the subintervals and determine the relative population level $p_i^t$ (the probability of a value from the sample falling into a subinterval i, that is, the relative frequency of occurrence of the “letter” in the “message”):

$$p_i^t=\frac{\Delta N_i}{N},\sum _{i=1}^n\Delta N_i=N,\sum _{i=1}^n{p}_i^t=1.$$

(11)

The elementary entropy of the sampling is defined as the Shannon entropy (9) on a given set $p_i^t$, and this is normalized to the total number of states n so that its values belong to the interval $[0,1]$:

$${\displaystyle H\left(p^t\right)=\frac{{\displaystyle -\sum _{i=1}^n{p}_i^t{log}_2{p}_i^t}}{{log}_{2}n}.}$$

(12)

This approach is known as the first sampling entropy [4] and is used, for example, in [24] to detect the hydroacoustic signals emitted by an underwater source.

On the other hand, the second sampling entropy can be defined as

$${\displaystyle H\left(p^0\right)=\frac{{\displaystyle -\sum _{i=1}^N{p}_i^0{log}_2{p}_i^0}}{{log}_{2}N},p_i^0=\frac{x\left(t_i\right)}{{\displaystyle \sum _{k=1}^{N}x\left(t_k\right)}}.}$$

(13)

In this case, the signal samples themselves are considered “letters”, which are distributed across the time axis in contrast to the amplitude axis from (12), and the “alphabet” is the whole set of amplitudes.

2.3. Spectral Entropy

In addition to the time domain, the entropy can be calculated based on the representation of the signal in the frequency domain, i.e., $p_i$ can be calculated with the spectrum of the signal. Spectral entropy is a quantitative assessment of the spectral complexity of the signal in the frequency domain from an energy point of view.

Consider the time series $x\left(t\right)$ and its spectral decomposition in the frequency domain $X\left(f_i\right)$ with $N_{fft}$ frequency components, obtained using the Fast Fourier Transform (FFT). The spectral power density is estimated as follows:

$$s\left(f_i\right)=\frac{1}{N_{fft}}X{\left(f_i\right)}^2.$$

(14)

Then, the probability distribution of the spectral power density $p^s=\{p_1,p_2,\dots ,p_{N_{fft}}\}$ can be written in the form

$${\displaystyle p_i^s=\frac{s\left(f_i\right)}{{\displaystyle \sum _{k=1}^{N_{fft}}s\left(f_k\right)}},i=1,\dots ,N_{fft},}$$

(15)

where $s\left(f_i\right)$ is the spectral energy for the spectral component with a frequency of $f_i$, $p_i^s$ is the corresponding probability density, $N_{fft}$ is the number of spectral components in the FFT, and the upper index s shows that the distribution refers to the signal spectrum. The resulting function is a spectrum distribution density function.

Finally, the spectral entropy can be determined with the Equation (9) and normalized by the size of the spectrum:

$${\displaystyle H\left(p^s\right)=\frac{{\displaystyle -\sum _{k=1}^{N_{fft}}{p}_k^s{log}_2{p}_k^s}}{{log}_2{N}_{fft}}.}$$

(16)

The Spectral Information Divergence ($SID$) method [25] was recently added to the Matlab mathematical package, and is calculated according to Formula (5) according to the similarity of two signals based on the divergence between the probability distributions of their spectra:

$$SID(r,t)=\sum _i{p}_{i}log\frac{p_i}{q_i}+\sum _i{q}_{i}log\frac{q_i}{p_i},$$

(17)

where r and t are the reference and test spectra, respectively, and the values of the probability distribution $p_i$ and $q_i$ for these spectra are determined according to (15).

3. Entropy Variation and Related Information Criteria

The purpose of this section is to determine the most appropriate formulas for calculating the information criteria that are responsible for the differences between distributions. Let us consider an entropy variation with respect to the variation in the probability distribution. The following lemma is valid:

Lemma 1.

For small variations $\delta q_i=p_i-q_i$ in the discrete distribution $q_i$, such as $p_i$, there is also some discrete distribution (10), and the decomposition of the entropy variation $\delta H$ in the case of series convergence by powers ${\displaystyle \delta q_i}$ has the form

$${\displaystyle \delta H=H(q+\delta q)-H\left(q\right)=LH\left(pq\right)-\frac{D(p,q)N}{2ln2}+o\left({\displaystyle {\left(\delta q\right)}^2}\right),}$$

(18)

The first summand of the entropy variation decomposition $\delta H$ is the difference between cross-entropy and entropy, and the second depends on the weighted squares of the variation of the distribution:

$$LH\left(pq\right)=H(p,q)-H\left(q\right),$$

(19)

$${\displaystyle D(p,q)=\frac{1}{N}\sum _{i=1}^N\frac{{\left(\delta q_i\right)}^2}{q_i}=\frac{1}{N}\sum _{i=1}^N\frac{{\left(p_i-q_i\right)}^2}{q_i}=\frac{1}{N}\left(\sum _{i=1}^N{q}_i{\left(\frac{p_i}{q_i}-1\right)}^2\right).}$$

(20)

The proof of the Lemma 1 is given in Appendix A.

Remark 1.

If q is the uniform distribution, i.e., $q_i=1/N$ for $i=1,...,N$, when

$$LH\left(pq\right)=H(p,q)-H\left(q\right)=0,$$

and the disequilibrium $D=D(p,q)$ is proportional to variance of the distribution p relative to the uniform one and is equal to

$${\displaystyle D_{SQ}(p,q)=\sum _{i=1}^N{\left(p_i-\frac{1}{N}\right)}^2=\sum _{i=1}^N{p}_i^2-\frac{1}{N}.}$$

(21)

Equation (21) coincides with the disequilibrium definition from [16].

According to Lemma 1 and Remark 1, we can introduce a new definition.

Definition 1.

In the case where q is the uniform distribution statistical complexity, as defined in [16], it is proportional to the first nonzero member of the row of square entropy variation, namely,

$$C_{SQ}=H\left(p\right)·D_{SQ}(p,q)=\left(-\sum _{i=1}^N{p}_i{log}_2{p}_i\right)·\left(\sum _{i=1}^N{\left(p_i-\frac{1}{N}\right)}^2\right).$$

(22)

Remark 2.

In general cases, the statistical complexity is defined as

$$C\sim \delta {\left(H\left(q\right)\right)}^2\sim H\left(q\right)\delta H\left(q\right)=H(H_{max}-H),$$

(23)

where $H_{max}$ is an entropy maximum.

It follows from Remark 1 that the disequilibrium (21) and the complexity (22) concelts must be applied when evaluating and comparing signals with background noise that has a spectral distribution close to uniform.

The formula for disequilibrium (21) proposed in [16] is derived from entropy variation, but most papers use the Jensen–Shannon divergence [22] for disequilibrium:

$$D_{JSD}=JSD\left(pq\right),$$

(24)

where $q_i=1/N$. The statistical complexity, correspondingly, is expressed as

$$C_{JSD}\left(p\right)=H\left(p\right)·JSD\left(pq\right).$$

(25)

Further on in the article, a comparison of the complexity graphs calculated for these two values $D_{SQ}$ and $D_{JSD}$ is presented.

Considering the signal distribution in the frequency and time domains, it can be observed that the spectral distribution does not require any additional estimation of the signal variance, whereas when calculating entropy in the time domain, variance estimation is required, since for white noise (Gauss distribution), the following formula is valid:

$$H_w=-{\int }_{-\infty }^{+\infty }{\rho }^t\left(x\right){log}_2\left({\rho }^t\left(x\right)\right)dx={log}_2\left(\sqrt{2\pi e}\sigma \right),$$

(26)

where ${\displaystyle {\rho }^t\left(x\right)=\frac{1}{\sqrt{2\pi }\sigma }exp(-{(x-\mu )}^2/\left(2{\sigma }^2\right))}$ is the Gaussian distribution.

Remark 3.

For the case of two continuous distributions $(p,q)$, the disequilibrium $D(p,q)$ is equivalent to the f-divergence [26] with the quadratic function f:

$${\displaystyle D(p,q)=\int {\rho }_q\left(x\right){\left(\frac{{\rho }_p\left(x\right)}{{\rho }_q\left(x\right)}\right)}^{2}dx,}$$

(27)

if an integral exists.

Remark 4.

In the case of two Gaussian distributions with the parameters $({\mu }_p,{\sigma }_p)$ and $({\mu }_q,{\sigma }_q)$, Formulas (19) and (20) from Lemma 1 take the form

$${\displaystyle D(p,q)=\frac{{\sigma }_q^2}{{\sigma }_p\sqrt{2{\sigma }_q^2-{\sigma }_p^2}}exp\left(\frac{{({\mu }_p-{\mu }_q)}^2}{2{\sigma }_q^2-{\sigma }_p^2}\right)-1,}$$

(28)

$${\displaystyle LH\left(pq\right)=H(p,q)-H\left(q\right)=\frac{{({\mu }_p-{\mu }_q)}^2}{2{\sigma }_q^2}+\frac{1}{2}\left(\frac{{\sigma }_p^2}{{\sigma }_q^2}-1\right).}$$

(29)

The Formulas (28) and (29) are obtained by calculating the integrals (27) and (26) for continuous distributions. These are also applicable for discrete ones.

The problem addressed here is the determination of the most informative methods for calculating entropy and other information criteria. In our opinion, the answer can only be obtained in the presence of additional knowledge about the phenomenon under study. Indeed, when measuring the amplitude of the signal $x\left(t\right)$, initially there is only knowledge of the time series samples $x\left(t_i\right)$, i.e., we know the values of amplitudes in the increasing sequence of time samples $t_i$, $i=1,\dots ,N$. Setting the distribution density using the Formula (11) itself allows some random variable to be defined.

Let us calculate the entropy $H\left(p^0\right)$ by applying grouping (11) and considering the fact that the entropy does not change as the summation order changes. Thus, the following chain of equations is valid:

$$\begin{array}{c}{\displaystyle H\left(p^0\right)=-\sum _{i=1}^N{p}_i^0{log}_2{p}_i^0\approx -\sum _{j=1}^n{p}_j^1{log}_2\frac{p_j^1}{\Delta N_j}={log}_{2}N+H\left(p^1\right)-H(p^1,p^t)}\\ {\displaystyle ={log}_{2}N-D_{KL}\left(p^1{p}^t\right),}\end{array}$$

(30)

where

$${\displaystyle p_j^1=\sum _{i\in I_j}{p}_i^0,I_j=\left\{i\in (1,\dots ,N):p_i^0\in \left[\frac{j-1}{n}\underset{k}{max}{p}_k^0,\frac{j}{n}\underset{k}{max}{p}_k^0\right]\right\},}$$

(31)

so that

$${\displaystyle p_j^t=\frac{\Delta N_j}{N},\Delta N_j=\sum _{i\in I_j}1.}$$

(32)

Thus, we have obtained that

$${\displaystyle H\left(p^0\right)\approx {log}_{2}N-D_{KL}\left(p^1{p}^t\right).}$$

(33)

If ${\displaystyle p_j^t=\frac{\Delta N_j}{N}=\frac{1}{n}}$ is now a uniform distribution, then (33) takes the form

$${\displaystyle H\left(p^0\right)={log}_{2}N-{log}_{2}n+H\left(p^1\right).}$$

(34)

The Formula (33) shows the relationship between the entropy calculated from the time samples and the Kullback–Leibler distance between the distributions obtained by alphabetical grouping along the amplitude and time coordinate axes. Therefore, the next statement is valid.

Corollary 1.

The value of $D_{KL}\left(p^1{p}^t\right)$ is approximately a constant value, independent of the method of grouping when the number of letters of the alphabet is large enough when their number is in some interval of values.

The following observation considering distribution $p^1$ is true.

Remark 5.

If the sequence of samples $x\left(t_i\right)$ has the property of ergodicity and the signal is represented by white noise, then if the number of letters of the alphabet from (31) is large enough, the density $p^1$ will be close to Gaussian.

Remark 5 allows us to estimate $H\left(p^0\right)$ using Equation (26) for the Gaussian $p^1$.

Now, there are four distribution densities at our disposal: three of them, $p^0$, $p^1$, and $p^t$, are related to the time domain and are determined by the Formulas (13), (31), and (32), and one, $p^s$, is related to the signal spectrum, calculated by the Formula (15).

With the presence of four distribution densities, the following criteria are considered simultaneously: the normalized information entropy H, defined by Formulas (12) and (16); the statistical complexity C, computed by (22) with a disequilibrium D (21); the Jenson–Shannon divergence $JSD$ (6); the spectral information divergence $\mathrm{SID}$ (17); and the cross-entropy and entropy difference $LH$ (19).

Since the spectral density is used to compare the signal/noise mixture with white noise, i.e., with a uniform distribution, all of the proposed criteria are applicable for this density. In the case of temporal distributions, the normalized information entropy H, which depends only on the distribution under study, and the difference of cross-entropy and entropy $LH$, calculated explicitly by the Formula (29) for ${\mu }_p={\mu }_q$, are estimated.

On the basis of the numerical experiments performed, a conclusion is made about the quality of the criteria used and the limits of their applicability in the presence of the noise component of the signal.

4. Hypothesis Testing

The classical probabilistic approach to the study of the considered problem of the detection of useful signals against background noise is called binary hypothesis testing. The binary problem associated with the decision to receive only noise (hypothesis ${\Gamma }_0$) or to receive a mixture of a useful signal and noise (hypothesis ${\Gamma }_1$) is solved [8].

In the statistical decision theory [6], it is shown that, in signal detection in the presence of noise, the optimal decisive rule is based on a comparison of the likelihood ratio with some threshold. The Neyman–Pearson criterion is used to select the threshold in the absence of a priori probabilities of the presence and absence of a useful signal. The efficiency of the detection procedure using the Neyman–Pearson criterion is characterized by the probability of correct detection with a fixed probability of false alarms.

The solution to the problem of distinguishing between two hypotheses can be derived from the following variant of the Neyman–Pearson lemma.

Lemma 2

(Neyman-Pearson). Let there be a measurable function, called a decisive rule,

$$d(x_1,\dots ,x_N)=\left\{\begin{array}{c}\hfill 1,the hypothesis{\Gamma }_0,\hfill \\ \hfill 0,the hypothesis{\Gamma }_1,\hfill \end{array}\right.$$

(35)

based on which

$$\alpha \left(d\right)=Probability(of detection{\Gamma }_0{\Gamma }_{1}is true),$$

$$\beta \left(d\right)=Probability(of detection{\Gamma }_1{\Gamma }_{0}is true).$$

The decisive rule $d^{*}$ is optimal if

$$\alpha \left(d^{*}\right)+\beta \left(d^{*}\right)=\underset{d}{inf}[\alpha \left(d\right)+\beta \left(d\right)]=\mathcal{E}r(N;{\Gamma }_0,{\Gamma }_1),$$

(36)

where $\mathcal{E}r(N;{\Gamma }_0,{\Gamma }_1)$ is called an error function.

In the problem of the detection of a useful signal, $\alpha$ is known as the probability of a false alarm occurring, and $\beta$ is known as the probability of missing a useful signal.

The error function can be calculated precisely through the variation of the measure (with a sign) by the following formula from [6]:

$${\displaystyle \mathcal{E}r(N;{\Gamma }_0,{\Gamma }_1)=1-\frac{1}{2}\parallel P_0^{\left(N\right)}-P_1^{\left(N\right)}\parallel =1-TV(P_0,P_1),}$$

(37)

where $P_0^{\left(N\right)}$ is the multivariate distribution function of observational statistics under hypothesis ${\Gamma }_0$, and $P_1^{\left(N\right)}$ is the multivariate distribution function of observational statistics under hypothesis ${\Gamma }_1$, and $TV$ is the full variation ${\displaystyle TV(P_0,P_1)=\frac{1}{2}\parallel P_0^{\left(N\right)}-P_1^{\left(N\right)}\parallel }$.

The peculiarity of the Formula (37) is that if the carriers on which the hypothesis measures ${\Gamma }_0$ and ${\Gamma }_1$ are concentrated are different, then $\mathcal{E}r(N;{\Gamma }_0,{\Gamma }_1)\approx 0$. If the measures $P_0^{\left(N\right)}$ and $P_1^{\left(N\right)}$ are similar, then $\parallel P_0^{\left(N\right)}-P_1^{\left(N\right)}\parallel \approx 0$, and then $\mathcal{E}r(N;{\Gamma }_0,{\Gamma }_1)\approx 1$.

For the problem of detecting a deterministic useful signal, the case $\parallel P_0^{\left(N\right)}-P_1^{\left(N\right)}\parallel =2TV(P_0,P_1)\approx 0$ and the possibility of a reasonable estimate of this value is interesting.

The estimated $TV(P,Q)$ constraints are known from thed estimate $\sqrt{JSD\left(PQ\right)}$, which is used to compute the statistical complexity $C_{JSD}$. Both $TV$ and $\sqrt{JSD}$ are metrics related to the probability distribution space, but in the Euclidean space, $\sqrt{D_{SQ}}$ (21) serves as this metric. Since the problem of detecting a deterministic signal in the presence of background noise is considered, it is reasonable to additionally take into account this ”determinism” by multiplying $D_{SQ}$ by the entropy H, which is associated with the introduction and use of statistical complexity in the form of (22) and (25).

5. Modelling and Discussion

5.1. The Calculation Algorithm and Presentation of the Simulation Results

In all experiments, graphs of the information characteristics are presented as functions of time. The characteristics are calculated from the signal according to the following algorithm:

• After being digitized with the sampling rate, the $\mathrm{F}$ audio signal is divided into short segments containing W digital samples.
• The discrete densities $p_i$ (15) are calculated from the time or frequency domains.
• The information criterion is calculated using $p_i$.
• The sequence of values is displayed together with the signal on the time axis (each of the obtained values is extended by W counts).

When a certain threshold of the information criterion is exceeded, this indicates the appearance of a useful signal in the mixture.

The signal processing results according to this algorithm are presented below. For different acoustic signals, a comparison of the quality of indication of the appearance of a useful signal by different information criteria at different levels of added white noise is demonstrated. In addition, Section 5.5 shows a comparison of two methods for calculating the statistical complexity and draws conclusions about the usefulness of both.

The first acoustic signal chosen was an audio recording of a humpback whale song recorded underwater. A large set of such recordings is available from the Watkins Marine Mammal Sound Database collected by Woods Hole Oceanographic Institution and the New Bedford Whaling Museum. The ability to separate such signals from strong sea noise may be useful for research biologists for further classification and study. In addition, these signals are similar in structure to the human voice with separate words, the extraction of which could be useful, for example, in tasks of voice activity detection and speech recognition.

In all of the graphs presented below, the signal is marked with a blue line, and the corresponding information metric is marked with a red line. The left vertical axis corresponds to the values of the signal amplitude, and the right vertical axis corresponds to the values of the information metric. All horizontal axes represent the timeline in seconds. The signal is shown without added noise for better comprehension, but the variable parameter of the standard deviation ${\sigma }_N$ of the white noise is marked with a dashed line. All information metrics are normalized for the convenience of presentation. All calculations and visualizations were performed using Python. White, brown, and pink noises, which were artificially added to audio recordings, were also generated numerically.

5.2. Time Information Criteria

First, we consider the behavior of the information entropies $H\left(p^t\right)$ and $H\left(p^0\right)$, calculated from the time samples of the signal $x\left(t\right)$.

Figure 1 shows that as noise increases, there is serious degradation of the time entropy graph for both calculation methods, so that for ${\sigma }_N=2000$ (SNR ≈ 1.5 dB), these information criteria can no longer serve as reliable indicators of the appearance of a useful signal in the mixture. We note an interesting feature of the behavior of $H\left(p^0\right)$ and $H\left(p^t\right)$: the value of the first characteristic is maximal for the uniform distribution and decreases with the appearance of a useful signal in the mixture, while, in contrast, the value of the second is minimal in the absence of a signal and increases with its appearance. This obviously follows on from the formulas for calculating the distributions and entropies (11), (13), (12).

Figure 1. Graphs of $H\left(p^t\right)$ and $H\left(p^0\right)$ for different levels of added noise.

The information characteristic $LH\left(pq\right)$ stands out favorably from the time entropies, as demonstrated above.

In Figure 2, the $LH$ for the noise level ${\sigma }_N=2000$ shows the appearance of a useful signal and works sufficiently, even for the double noise value. However, it should be noted that this is true only for stationary noise, whose average value does not change over time. Otherwise, this metric will react to changes in noise as well, which follows on from the formula (29). Moreover, initial estimation of ${\sigma }_N$ is required for the correct functioning of this criterion.

Figure 2. Graphs of $LH\left(pq\right)$ for different levels of added noise.

5.3. Time Entropy $H\left(p^1\right)$

The time entropy $H\left(p^1\right)$ associated with another grouping of the ”alphabet” derived from the signal samples is considered separately and the graphs for different number of letters are shown in Figure 3 and Figure 4.

Figure 3. The entropy $H\left(p^1\right)$ for the number of letters of the alphabet equal to 64.

Changing the alphabet partitioning negatively affects the effectiveness of entropy in this representation:

Figure 4. The entropy $H\left(p^1\right)$ for the number of letters of the alphabet equal to 8.

5.4. Spectral Information Criteria

Information criteria based on the spectral distribution of $p^s$ are deprived of the disadvantages of the time criteria.

Figure 5 shows the dependence of the spectral entropy on time. We can see a significant improvement in the maximum allowable noise level, at which the indication of the appearance of a useful signal is still possible, with respect to the graphs presented in Figure 2.

Figure 5. Spectral entropy plots for different SNRs.

The point that we want to make is that the white noise in a signal in spectral representation has quite a definite uniform probability distribution, which greatly facilitates the calculation of entropy and saves us from the necessity of estimating the variance of this noise. Moreover, even if the noise is not stationary, i.e., its parameters change over time, in a small window W, it can still be considered white, and the above statement is still true.

The distribution $p^s$ can be used as the basis for a number of information divergences (17), (24), (22), (25):

Figure 6 shows that the separability of information metrics decreases along with the signal-to-noise ratio (SNR). However, the statistical complexity $C_{SQ}$ performs better than all other criteria, because it still allows a useful signal to be distinguished when other metrics behave irregularly and no longer show significantly excess levels compared to areas without a signal. Thus, it is the most promising characteristic in our opinion.

Figure 6. Information divergence plots based on spectral distribution.

5.5. Comparison of Different Ways of Calculating the Statistical Complexity

Of separate interest is the comparison of the behavior of the statistical complexities $C_{SQ}$ and $C_{JSD}$, which essentially correspond to different methods of calculating the same value of statistical complexity. Figure 7 illustrates this comparison.

Figure 7. Comparison of statistical complexities $C_{SQ}$ and $C_{JSD}$.

We can see that $C_{SQ}$ shows a better result when used as an indicator of the appearance of a useful signal in white noise compared to the Jensen–Shannon divergence.

5.6. Hydroacoustic Signal Model of an Underwater Marine Object

The second signal is a modelled hydroacoustic signal of an underwater marine object. The study of such signals is important in military and civilian applications, because it can automate the process of analyzing the hydroacoustic scene and identifying potential threats. In Figure 8, spectral entropy dependencies for different levels of added noise are shown.

Figure 8. Spectral entropy plots for different SNRs.

Figure 9 shows the dependencies of statistical complexity for a given signal. It is worth noting that the selected information metric shows the presence of a useful signal, even for a very small SNR ($\approx -17$ dB) in the last example.

Figure 9. Graphs of statistical complexity for different SNRs.

It can be observed that, in comparison with all other information metrics, the statistical complexity shows the best result in terms of indicating the presence of a useful signal in the mixture, because it remains effective for small SNRs, while all other characteristics can no longer detect a useful signal in noisy receiving channels.

5.7. Hydroacoustic Signal Model with Pink Noise

Now let us change the additive noise model and use pink noise instead of white noise. As can be observed in Figure 10, the spectral entropy shows an unsatisfactory result for the chosen low SNR.

Figure 10. Spectral entropy for pink noise model.

Figure 11 shows that, along with the spectral entropy, the statistical complexity $C_{SQ}$ performs poorly, but $C_{JSD}$ confidently shows the presence of a signal.

Figure 11. Statistical complexities $C_{SQ}$ and $C_{JSD}$ for pink noise model.

5.8. Hydroacoustic Signal Model with Brown Noise

In this example, brown noise is used as the noise model. As in the previous subsection, spectral entropy fails in the task of signal extraction, as shown in Figure 12.

Figure 12. Spectral entropy for brown noise model.

However, Figure 13 shows that the statistical complexity with the Jensen–Shannon disbalance exhibits a satisfactory performance.

Figure 13. Statistical complexities $C_{SQ}$ and $C_{JSD}$ for brown noise model.

The results are summarized in Table 1. The checkmark indicates the possibility of confident indication of the useful signal, and x indicates the lack of this.

Table 1. Comparison of information criteria for different noise models and SNRs.

Noise Type	SNR (dB)	Entropy $\mathit{H}\left({\mathit{p}}^{\mathit{s}}\right)$	Statistical Complexity ${\mathit{C}}_{\mathit{S}\mathit{Q}}$	Statistical Complexity ${\mathit{C}}_{\mathit{J}\mathit{S}\mathit{D}}$
White	−10	✓	✓	✓
	−17	×	✓	×
Pink	−5	×	✓	✓
	−10	×	×	✓
Brown	−5	×	✓	✓
	−10	×	×	✓

6. Conclusions

The article proposed a method for indicating the appearance of a useful signal in a heavily noisy mixture based on the statistical complexity. The analytical formulas used to determine the disequilibrium and statistical complexity were obtained using entropy variation. The effectiveness of the proposed approach for two types of acoustic signals in comparison with other information metrics was shown for different models of added noise. For white noise, the appearance of deterministic signal was shown to be reliably detected for a very small SNR ($\approx -15$ dB) when the statistical complexity based on the spectral distribution variance was used as the criterion. However, for more complex noise models, the use of the statistical complexity with the Jensen–Shannon disequilibrium was shown to have better efficiency. Both the time and frequency domains were considered for the entropy calculation. The criteria for signal detection in a heavy noise mixture based on time distributions were shown to be less informative than those based on spectral distribution. The connection between the statistical complexity criterion and the Neyman–Pearson approach for hypothesis testing was also discussed. Future work will be devoted to research on the information criteria based on two- and multidimensional distributions, and acoustic signals with realistic background noise will be considered.