An Entropy Measure of Non-Stationary Processes

Shannon’s source entropy formula is not appropriate to measure the uncertainty of non-stationary processes. In this paper, we propose a new entropy measure for non-stationary processes, which is greater than or equal to Shannon’s source entropy. The maximum entropy of the non-stationary process has been considered, and it can be used as a design guideline in cryptography.


Introduction
Information science considers an information process, uses the probability measure for random states and Shannon's entropy as the uncertainty function of these states [1][2][3].For a stationary source, the probability and the Shannon's entropy can be determined by using statistical physics methods, while for a non-stationary source, all the statistical physics methods fail, thus the Shannon entropy measure may not be available.Till now, the entropy of non-stationary process is still not fully understood, except for some specific types of non-stationary process [4] that use the following entropy formula to measure the uncertainty: ( , )ln ( , ) H P x t P x t dx This kind of non-stationary process requires a known probability function, and the probability function is deterministically varied.This formula will fail if the probability function of the non-stationary process is stochastically varied.For a non-stationary process, if its parametric is a OPEN ACCESS stationary random variable, then this kind of non-stationary process can be considered as a piecewise stationary process, and many papers use the following source entropy formula, proposed by Shannon, to measure the uncertainty [5][6][7][8][9]: , ( )log ( ) where for each possible state i there will be a set of probabilities p i (j) of producing the various possible symbols j.
In this paper, we will consider the non-stationary process, with its parametric be a stationary random variable.We will show that the entropy of this kind of non-stationary process should not be measured by Shannon's source entropy (Equation ( 2)).Actually, Shannon's source entropy Equation ( 2) is only used for a stationary process with multiple-states, not for non-stationary processes.In our paper, we will propose an entropy formula to measure the uncertainty of non-stationary processes.Our entropy measure is greater than or equal to Shannon's source entropy formula.These two measures are equal if and only if the process is stationary.Our entropy measure can be used as a complexity criterion of non-stationary processes and as a design guideline for cryptographic uses.
The rest of this paper is organized as follows: Section 2 shows that Shannon's source entropy formula is not appropriate to measure the uncertainty of non-stationary processes.In Section 3, a new entropy formula is proposed to measure the uncertainty of non-stationary processes and some properties are presented.In Section 4, the maximum entropy of the non-stationary process is considered.Section 5 concludes the paper.

The Limitation of Shannon's Source Entropy in Non-Stationary Process
First, let's consider the following two examples, one is for discrete source and the other is for continuous source: Example 1.Consider a discrete source of two kinds of symbol 0 and 1.This discrete source varies between the states "1" and "2" randomly with probabilities P 1 = 0.2 and P 2 = 0.8.For state "1", the probability of producing the symbol 0 is 0.4, and producing the symbol 1 is 0.6.For state "2", the probability of producing the symbol 0 is 0.3, and producing the symbol 1 is 0.7.Obviously, this discrete source is non-stationary.By using Shannon's source entropy Equation (2), we have H = 0.6233.By calculating the entropy of the 01 sequences which generated by this discrete source, we have H approaches to 0.629 with the sequence length increased, which is bigger than Shannon's source entropy measure.
Example 2. Consider a continuous source of a parameter-varied normal distribution N(μ, 1), which parameter μ varies between μ 1 , μ 2 , …, μ n randomly with probabilities p 1 , p 2 , … ,p n .Obviously, this continuous source is non-stationary.By using Shannon's source entropy Equation (2), we have H = 0.5ln2πe, which is not relevant to μ.Then, the uncertainty of this non-stationary source equals to the uncertainty of normal distribution source with fixed μ as N(0, 1), which is not convinced (the variation of parameter μ also brings some of uncertainty when determining the states).
Therefore, Shannon's source entropy formula is not appropriate to measure the uncertainty of a non-stationary process.The entropy of non-stationary processes is always bigger than Shannon's source entropy measure.Next, we will analyze why this happens.
Let s 1 s 2 …s i(1) s i(1)+1 …s i(2) s i(2)+1 …s N be a symbol sequence generated by the source of Example 3, which s 1 s 2 …s i( 1) is generated by a fixed random parameter, s i(1)+1 …s i( 2) is generated by a fixed random parameter, and so on.For the first symbol s 1 , its uncertainty can be written as h(p) o H 1 .Here, h(p) is the uncertainty of varying parameter, and H 1 is the uncertainty of symbol s 1 with a known parameter."o" is an operator which has the following properties: For the symbol s 2 , its uncertainty is H 2 , without h(p) for its parameter is determined.For the symbol s i(1)+1 , its uncertainty is h(p) o H i(1)+1 , ….Then the average uncertainty of this non-stationary source can be written as: here, E|s i(j)+1 …s i(j+1) | is the average length of symbol sequence by each parameter.The "=" holds if and only if the source is stationary, which degenerate to Shannon's source entropy measure.

The Entropy of Non-Stationary Process and Its Properties
According to the Boltzmann-Gibbs theorem, we should consider all the possible states and their probabilities when we measure the uncertainty of a source.Consider a non-stationary process which varies randomly in n kinds of possible states.The probabilities of each states are p 1 , p 2 , …, and p n respectively.For the state i, the output variable satisfies the probability density function f i .The probability of generating a N-length sequence by this non-stationary source is: Then, the uncertainty of the N-length sequence is: The average uncertainty of this non-stationary process is: Theorem 1.
Proof.First, we consider the case N = 2.We have: Assume the equation holds when N = k.Then for N = k + 1, denote 1 ( ) , then we have: Thus, concluding our proof.According to Equations ( 5) and ( 6) and theorem 1, the entropy of the non-stationary process can be written as: ( ) log( ) Equation ( 7) is the entropy formula of the non-stationary process when the states vary discretely.For a continuously case, the entropy formula can be written as:

( ( ) ) log( ( ) )
y y H g y f dy g y f dydx Here, g(y) is the probability density function of the varying state and f y is the probability density function of the output variable for each state y.We have that by using the entropy equations ( 7) and ( 8) instead of Shannon's source entropy, there are no such inconsistencies as proposed in examples 1 and 2. Theorem 2. Let H S and H be the Shannon's entropy measure and our entropy measure of a non-stationary process respectively, H(p) be the Shannon's entropy of the varying states.We have the following inequality hold: ( )

H H H H p   
Proof.We use Equation ( 7) as our entropy formula (states vary discretely).For the continuous case, it can be proven similarly.
We consider H -H s , and have: Thus, we have H < H s + H(p).Additionally, consider the following functional F: log( ) It is easily to prove that functional F reaches its minimum when f 1 = f 2 = … = f n , we have: Thus, we have H ≥ H s , which concluding our proof.Theorem 2 shows that the entropy of non-stationary process is greater than or equal to the Shannon's entropy measure.These two measures are equal if and only if the process is stationary.Furthermore, H < H s + H(p) means that the entropy of non-stationary process cannot be written as the sum of H s and H(p), although its uncertainty is brought by these the aspects.In another word, they are not independent of each other.

Maximum Entropy and Its Application in Cryptography
In recent years, chaotic systems were regarded as an important pseudorandom source in the design of random number generators [10][11][12].As we know, chaotic systems may be attacked by phase space reconstruction and nonlinear prediction techniques.If the system is abiding references [13][14][15] propose a chaotic system with varying parameters to resist these attacks.However, the varying method is rather simple.A more secure method is to vary the parameters in a random-like way, then the output sequence come to be non-stationary.Entropy is an important criterion in cryptographic use.With our entropy formula of non-stationary process, we can compare the different varying methods and guide us to design a varying method in order to make the entropy maximum.It can be expressed as the following mathematical problem.

P1.
The maximum of functional: ( ) log( ) Design a Lagrange functional: Solve the following equations: Compare these extreme values in order to get the maximum entropy and the corresponding p i .Consider the discrete case (the output variables of each states are discrete).Assume the non-stationary process is varying in n kinds of possible states a 1 , a 2 , … , a n with probabilities p 1 , p 2 , … , p n .For each state a i , the output variables are b 1 , b 2 , … , b N and the probabilities are f i1 , f i2 , … , f iN respectively.Then, the entropy of this non-stationary process is: By using the above Lagrange method, we have that the entropy (9) reach its maximum value when the following equation holds: The maximum entropy is logN which equals to the stationary uniform distribution.
For the continuous case (the output variables of each states are continuous), the method is similar.Next, we show two simple examples.
Example 3. Consider a non-stationary process which varies between two possible states a 1 and a 2 with probabilities p 1 and p 2 respectively.For state a 1 , the output variables are b 1 and b 2 with probabilities f 11 = 0.75 and f 12 = 0.25.For state a 2 , the output variables are b 1 and b 2 with probabilities f 21 = 1/3 and f 22 = 2/3.By solving the maximum entropy problem, we have that the entropy reaches its maximum value when p 1 = 0.4, p 2 = 0.6, and H max = log2.
Example 4. Consider a non-stationary process which varies in two states a 1 and a 2 with probabilities p 1 and p 2 respectively.For state a 1 , the output variables satisfy the normal distribution N(0, 1).For state a 2 , the output variables satisfy the normal distribution N(1, 1).By solving the maximum entropy problem, we have that the entropy reaches its maximum value when p 1 = 0.5, and H max = 1.5304.Figure 1 shows the relation between the Entropy and p 1 .

Conclusions
In this paper, we first prove that Shannon's source entropy is not appropriate to measure the uncertainty of non-stationary processes.Then, we propose an entropy formula to measure the uncertainty of such non-stationary processes.Our entropy measure is greater than or equal to Shannon's source entropy formula.These two measures are equal if and only if the process is stationary.Furthermore, we study the maximum entropy of our entropy formula.The maximum entropy can be used as a guideline for constructing a non-stationary source in cryptographic uses.

Figure 1 .
Figure 1.The relation between the entropy and p 1 .