Low-Entropy Stochastic Processes for Generating k-Distributed and Normal Sequences, and the Relationship of These Processes with Random Number Generators †

Abstract

An infinite sequence $x_1{x}_2\dots$ of letters from some alphabet $\{0,1,$$\dots ,b-1\}$, $b\ge 2$, is called k-distributed ($k\ge 1$) if any k-letter block of successive digits appears with the frequency $b^{-k}$ in the long run. The sequence is called normal (or ∞-distributed) if it is k-distributed for any $k\ge 1$. We describe two classes of low-entropy processes that with probability 1 generate either k-distributed sequences or ∞-distributed sequences. Then, we show how those processes can be used for building random number generators whose outputs are either k-distributed or ∞-distributed. Thus, these generators have statistical properties that are mathematically proven.

1. Introduction

In 1909, Borel defined k-distributed and ∞-distributed sequences as follows: A sequence of digits in base b is k-distributed if for any k-letter word w over the alphabet $\{0,1,$$\dots$$,b-1\}$

$$\underset{t\to \infty }{lim}{\nu }_t\left(w\right)/(t-|w\left|\right)=b^{-\left|w\right|}$$

(1)

where ${\nu }_t\left(w\right)$ is a number of occurrences of w in the sequence $x_1\dots x_{\left|w\right|}$, $x_2\dots x_{\left|w\right|+1},\dots$, $x_{t-\left|w\right|+1}\dots x_t$). The sequence is normal (or ∞-distributed) if it is k-distributed for any $k\ge 1$. Borel called normal to base b a real number from the interval $(0,1)$ whose expansion in base b is normal sequence, and showed that almost all real numbers are normal to any base (with respect to the uniform measure) [1,2]. It is interesting that the construction of ∞-distributed sequence in an explicit form was first achieved by Champernowne in 1933 [3], who proved that the sequence

$$012\dots 9101112\dots 99100101102\dots$$

is ∞-distributed. Later, many ∞-distributed sequences were described and investigated in numerous papers (see for review [4]). Many researchers suppose that fractional parts of $\pi$, e, and $\sqrt{2}$ and some other “mathematical” constants are normal, but it is not proven [2,5]. On the other hand, for $\pi$ empirical counting over several billions of its digits suggests that this might be true (see [5,6]).

One of the reasons for interest in k- and ∞-distributed sequences is due to the fact that they are closely related to the concept of randomness. If we imagine that someone tosses a fair coin with sides marked 0 and 1, he/she obtains (almost surely) an ∞-distributed sequence [2,5]. A mathematical model of such an experiment is a sequence of an independent and identically distributed (i.i.d.) symbols from $\{0,1\}$ generated with probabilities $(1/2,1/2)$. Note that quite often this i.i.d. process and the sequences generated from them are called “true random” [2].

The true random sequences are very desirable in cryptography, simulation and modeling applications. Of course, it is practically impossible to generate them tossing a coin and nowadays there are many so-called generators of pseudo-random numbers (PRNGs), whose aim is, informally speaking, to calculate sequences which mimic the truly random (see [2,7,8,9,10]). For brevity, in what follows, we consider the case when a process generates letters from the alphabet $\{0,1\}$, but the obtained results can be extended to the case of any alphabet.

Modern PRNGs is a computer program whose input is a short word (a so-called seed), whereas its output is a long (compared to the input) word. Having taken into account that the seed is a true random word, the PRNG can be considered as an expander of randomness which stretches a short seed into a long word [2,7,10]. The output of “perfect” PRNG would have to generate true random output sequence. However, it is impossible.

To be more precise, we note that a mathematically correct definition of a random sequence was obtained in a framework of algorithmic information theory established by Kolmogorov (see [11,12,13,14,15]). In particular, it is shown that any algorithm (i.e., a Turing machine) can neither generate (infinite) random sequences nor stretch a short random sequence into a longer one. It means that PRNGs do not exist. The same is true in a framework of Shannon information theory. Indeed, it is known that the Shannon entropy of the true random process (i.e., i.i.d. with probabilities $(1/2,1/2)$) is one bit per letter, whereas for all other processes the entropy (per letter) is less than one (see [16]). On the other hand, any PRNG stretches a short true random sequence into a long one. The entropy of the output is not greater then the entropy of the input and, hence, the per letter entropy of the output is strictly less than 1 bit. Therefore, the demands of true randomness and low entropy are contradictory. Thus, we see that, in a framework of algorithmic information theory, as well as in a framework of Shannon information theory, “perfect” PRNGs do not exist.

In such a situation, researchers suggest and investigate PRNGs, which meet some “probabilistic” properties of true random sequences [2,17]. In particular, a property that a PRNG generates ∞-distributed sequences is very desirable (see [2]).

Another important type of random number generators (RNGs) is physical random number generators, among which the so-called quantum random number generators (QRNG) have become very popular in recent decades and are widely used in practice. By definition, the physical RNGs are devices whose output is a binary sequence that must be truly random (or at least look truly random) (see [10]). According to M. Herrero-Collantes and J.C. Garcia-Escartin [10], a physical RNG can be divided into the two following blocks: the entropy source and the post-processing stage. The output of the source of entropy is a bit string obtained by measuring a physical random process with subsequent quantization. The goal of post-processing is to translate this bit string into a true random binary sequence. Nowadays, there are many methods of post-processing [10], but, nevertheless, the statistical (probabilistic) properties of many physical RNGs and, in particular, QRNGs, are not proven mathematically and should be experimentally tested [10,18,19]. Even the so-called device-independent QNRGs guarantee only the randomness of their output, but true randomness must either be verified or obtained by post-processing [10]. Thus, transformations that transform the output into a normal sequence are desirable for all types of RNGs.

Here, we describe such random processes that their entropy is much less than 1, but Equation (1) is valid for generated sequences either for all integers or for k from a interval $1,\dots ,K$, where K is an integer. This shows that there exist low-entropy PRNGs which generate such sequences that Equation (1) is valid (for $b=2$). The description of the suggested processes show that they can be used to develop PRNGs with the property in Equation (1). The described processes are generalization of so-called two-faced processes suggested in [20,21,22].

In detail, we propose the following two processes. First, we describe the k-order Markov chain, which is a so-called two-faced process of order k, $k\ge 1$, for which, with probability one, for any generated sequence $x_1{x}_2\dots$ and all binary words $w\in {\{0,1\}}^k$, the frequency of occurrence of the word w in the sequence $x_1\dots x_{\left|w\right|}$, $x_2\dots x_{\left|w\right|+1},\dots$, $x_{t-\left|w\right|+1}\dots x_t\dots$ goes to $2^{-\left|w\right|}$. Secondly, we describe so-called normal two-faced processes for which this property is true for all k.

We also propose the so-called two-faced transformation, which translates the trajectories of any random process into the trajectories of a two-faced process. This transformation is applicable to the creation of a PRNG with proven statistical properties.

2. K-Distributed Sequences and Two-Faced Markov Chains

First, we consider a pair of examples in order to explain the main idea of considered Markov chains. Let a matrix of transition probabilities $T^{\prime }$ be as follows:

$$\begin{array}{ccc}\hfill & 0& 1\hfill \\ \hfill 0& {\alpha }_0& {\alpha }_1\hfill \\ \hfill 1& {\alpha }_1& {\alpha }_0\hfill \end{array},$$

(2)

where ${\alpha }_0$ and ${\alpha }_1$ are non-negative and their sum equals 1 (i.e., $P\{x_{i+1}=0|x_i=0\}={\alpha }_0$, $P\{x_{i+1}=0|x_i=1\}={\alpha }_1,\dots$).

For example, let ${\alpha }_0=0.9,{\alpha }_1=0.1$. Then, the “typical” output sequence can be as follows:

$$0000000000111111111000000000011111110\dots .$$

On the one hand, this sequence is clearly not true random. On the other hand, the frequencies of 1s and 0s goes to $1/2$ due to the symmetry of the matrix in Equation (2). Hence, the output is 1-distributed. Again, based on the symmetry, we can build the following matrix of the second order whose output will be 2-distributed:

$$\begin{array}{ccccc}\hfill & 00& 01\hfill & 10& 11\\ \hfill 0& {\alpha }_0& {\alpha }_1\hfill & {\alpha }_1& {\alpha }_0\\ \hfill 1& {\alpha }_1& {\alpha }_0\hfill & {\alpha }_0& {\alpha }_1\end{array}$$

(3)

(Here, $P\{x_{i+1}=0|x_i=0,x_{i-1}=0\}={\alpha }_0$, $P\{x_{i+1}=0|x_i=0,x_{i-1}=1\}={\alpha }_1,\dots$.) For ${\alpha }_0=0.9,{\alpha }_1=0.1$, the “typical” output sequence can be as follows:

$$000000000000110110110110110110110110110000\dots ,$$

where gaps correspond to seldom transitions. It can be easily seen that frequency of any two-letter word goes to $1/4$.

Let us give a formal definition of two-faced Markov chains. First, we define two families of random processes $T_{k,p}$ and ${\widehat{T}}_{k,p}$, where integer k and $p\in (0,1)$ are parameters. The processes $T_{k,p}$ and ${\widehat{T}}_{k,p}$ are Markov chains of the connectivity (memory) k, which generate letters from the binary alphabet. We define them inductively. The matrix of $T_{k,p}$ is defined as follows: $T_{1,p}(0,0)=p,T_{1,p}(0,1)=1-p$, $T_{1,p}(1,0)=1-p,T_{1,p}(1,1)=p$. The process ${\widehat{T}}_{1,\pi }$ is defined by ${\widehat{T}}_{1,p}(0,0)=1-p,{\widehat{T}}_{1,p}(0,1)=p$, ${\widehat{T}}_{1,p}(1,0)=p,{\widehat{T}}_{1,p}(1,1)=1-p$. Let the transition matrices $T_{k,p}$ and ${\overline{T}}_{k,p}$ be defined, then $T_{k+1,p}$ and ${\widehat{T}}_{k+1,p}$ are as follows

$$T_{k+1,p}(0,0u)=T_{k,p}(0,u),$$

(4)

$$T_{k+1,p}(1,0u)=T_{k,p}(1,u),$$

$$T_{k+1,p}(0,1u)={\widehat{T}}_{k,p}(0,u),$$

$$T_{k+1,p}(1,1u)={\widehat{T}}_{k,p}(1,u),$$

Conversely,

$${\widehat{T}}_{k+1,p}(0,0u)={\widehat{T}}_{k,p}(0,u),$$

(5)

$${\widehat{T}}_{k+1,p}(1,0u)={\widehat{T}}_{k,p}(1,u),$$

$${\widehat{T}}_{k+1,p}(0,1u)=T_{k,p}(0,u),$$

$${\widehat{T}}_{k+1,p}(1,1u)=T_{k,p}(1,u)$$

for all $u\in {\{0,1\}}^k$ ( $vu$ is a concatenation of v and u). We can see that

$$T_{k+1}=\left(T_{k,}{\widehat{T}}_k\right).$$

(6)

For example,

$$T_{2,p}(0,00)=p,T_{2,p}(0,01)=1-p,T_{2,p}(0,10)=1-p,T_{2,p}(0,11)=p.$$

To describe the process, the initial probability distribution should be defined. We say that the initial distribution of $T_{k,\pi }$ and ${\widehat{T}}_{k,\pi }$ is uniform, if for all $w\in {\{0,1\}}^k$$P\{x_1\dots x_k=w\}=2^{-k}$. Sometimes, we consider different initial distributions, which is why, in all cases, the initial distribution is mentioned.

Let $\mu$ be stationary process. Its conditional Shannon entropy of order m, $m=1,2,\dots$, is defined as follows

$$h_m=-\sum _{w\in {\{0,1\}}^{m-1}}\mu \left(w\right)\sum _{u\in \{0,1\}}\mu (u/w)log\mu (u/w)$$

(7)

and the limit entropy is as follows

$$h_{\infty }=\underset{r\to \infty }{lim}{h}_r,$$

(8)

see [16].

The main properties of Markov chains $T_{k,p}$ and ${\widehat{T}}_{k,p}$, $k\ge 1$, are described by the following

Theorem 1.

Let $x_1{x}_2\dots$ be generated by $T_{k,p}$ (or ${\widehat{T}}_{k,p}$), $k\ge 1$, and w $\in {\{0,1\}}^k$. Then,

(i) 

If the initial distribution is uniform over ${\{0,1\}}^k$, then

$$P(x_{j+1}\dots x_{j+k}=w)=2^{-\left|w\right|}$$

(9)

for any $j\ge 0$.

(ii) 

For any initial distribution of the Markov chain $T_{k,p}$ (or ${\widehat{T}}_{k,p}$)

$$\underset{j\to \infty }{lim}P(x_{j+1}\dots x_{j+k}=w)=2^{-\left|w\right|}.$$

(10)

(iii) 

With probability one the Markov chains $T_{k,p}$ and ${\widehat{T}}_{k,p}$ generates k-distributed sequences.

(iv) 

For any $p\in (0,1)$, the k-order Shannon entropy ($h_k$) of the processes $T_{k,p}$ ( ${\widehat{T}}_{k,p}$) equals 1 bit per letter, whereas the limit entropy equals $-(p{log}_{2}p+(1-p){log}_2(1-p)).$

The proof is given in Appendix A.

Having taken into account this theorem, we give the following.

Definition 1.

If Equation (10) is valid for any $w\in {\{0,1\}}^k$, the process is asymptotically two-faced of order k. The process is two-faced of order k, if Equation (9) is true.

It turns out that, in a certain sense, there are many two-faced processes. More precisely, the following theorem is true.

Theorem 2.

Let $X=x_1{x}_2\dots$, $Y=y_1{y}_2\dots$ be random processes. We define the process $Z=z_1{z}_2\dots$ by following equations $z_1=x_1\oplus y_1$, $z_2=x_2\oplus y_2,\dots$ where $a\oplus b=(a+b)mod2$. If X is a k-order (asymptotically) two-faced process, then Z is also the k-order (asymptotically) two-faced process ($k\ge 1$).

The proof is given in Appendix A.

3. Two-Faced Transformation

Now, we show that any stochastic process can be transferred into a two-faced one. For this purpose, we describe transformations which transfer random processes into two-faced ones. First, we define matrices $M_k$ and ${\widehat{M}}_k$, $k\ge 1$, which are based on matrices $T_{k,p}$ and ${\widehat{T}}_{k,p}$.

Definition 2.

The matrix $M_k$ is defined by the following equation:

Let us define the matrix $M_k$ as follows:

$$M_k(w,v)=\left\{\begin{array}{c}0,if{T}_{k,p}(w,v)=p\\ 1,if{T}_{k,p}(w,v)=1-p\end{array}\right.$$

(11)

for any $k\ge 1$, $v\in {\{0,1\}}^k$, $w\in \{0,1\}$. The matrix ${\widehat{M}}_k$ is obtained from ${\widehat{T}}_{k,p}$ analogously. Note that, from Equation (6), we obtain

$$M_{k+1}=\left(M_k{\widehat{M}}_k\right).$$

(12)

Definition 3.

Let k be an integer, $v_1\dots v_k\in {\{0,1\}}^k$, $u\in \{0,1\}$. Define functions ${\tau }_k$ and ${\overline{\tau }}_k$ as follows:

$${\tau }_k(u,v_1\dots v_k)=M_k(u,v_1\dots v_k),{\overline{\tau }}_k(u,v_1\dots v_k)={\widehat{M}}_k(u,v_1\dots v_k).$$

(13)

Let $X=x_1{x}_2\dots$, and $v\in {\{0,1\}}^k$. The two-faced transformation ${\tau }_k$ maps a pair $(X,v)$ into a sequence Y as follows: $y_{-k+1}{y}_{-k+2}\dots y_0=v,$ $y_i={\tau }_k(x_i,y_{i-k}{y}_{i-k+1}\dots y_{i-1}),$ where $i=1,2,\dots$.

Note that, from this definition and Equation (12), we can see that for any $i\ge 1$

$${\tau }_k(x,y_{i-k+1}{y}_{i-k+2}\dots y_i)=\left\{\begin{array}{c}{\tau }_{k-1}(x,y_{i-k+2}\dots y_i),if{y}_{i-k+1}=0\\ {\overline{\tau }}_{k-1}(x,y_{i-k+2}\dots y_i),if{y}_{i-k+1}=1\end{array}\right.$$

(14)

Theorem 3.

Let $k\ge 1$ be an integer, $X=x_1{x}_2\dots$ be generated by a stochastic process, and ${\tau }_k$ be a two-faced transformation. If v is uniformly distributed on ${\{0,1\}}^k$, then for any $u\in {\{0,1\}}^k$ and $r\ge 1$

$$P\{y_{r-k}{y}_{r-k+1}\dots y_{r-1}=u\}=2^{-k},$$

(15)

i.e., ${\tau }_k(X,v)$ is two-faced of order k process. The proof is given in Appendix A.

Consider now the question of the complexity of the described transformation ${\tau }_k$ allowing transform any process into a two-faced. When directly implementing the transformation ${\tau }_k$, one must store matrix $M_k$ of 2 rows and $2^k$ columns, i.e., just $2^{k+1}$ numbers. Storing such matrices becomes impossible when k exceeds hundreds. Therefore, the question arises of constructing a simpler algorithm that does not require an exponential growth of memory with increasing k. It turns out that there exists an algorithm which requires $O\left(k\right)$ bits of memory and finite number of operation (per an output letter).

To describe this algorithm, we first define transformations ${\tau }_k^{*}$ and ${\overline{\tau }}_k^{*}$. Let there be an infinite word $x_1{x}_2\dots$ and a finite one $y_{-k+1}{y}_{-k+2}\dots y_0$. For any $r\ge 1$, denote $H_r=\left({\sum }_{r-k+1}^r{y}_i\right)mod2$$\overline{H}=H\oplus 1$. Then, ${\tau }_k^{*}(x_{r+1},y_{r-k+1}{y}_{r-k+2}\dots y_r)$$=H_r\oplus x_{r+1}$ and $y_{r+1}={\tau }_k^{*}(x_{r+1},y_{r-k+1}{y}_{r-k+2}\dots y_r)$. Similarly, ${\overline{\tau }}_k^{*}(x_{r+1},y_{r-k+1}{y}_{r-k+2}\dots y_r)$$={\overline{H}}_r\oplus x_{r+1}$. From those definitions and Equation (12), we can see that for any $i\ge 1$

$${\tau }_k^{*}(x,y_{i-k+1}{y}_{i-k+2}\dots y_i)=\left\{\begin{array}{c}{\tau }_{k-1}^{*}(x,y_{i-k+2}\dots y_i),if{y}_{i-k+1}=0\\ {\overline{\tau }}_{k-1}^{*}(x,y_{i-k+2}\dots y_i),if{y}_{i-k+1}=1\end{array}\right.$$

(16)

It is important to note that there exists the simple algorithm for carrying out the transformation ${\tau }_k^{*}$. Indeed, just store the letters $y_{r-k+1}{y}_{r-k+2}\dots y_r$ and the value $H_r$ in the computer’s memory. Then, read the letter $x_{r+1}$, calculate $y_{r+1}=H\oplus x_{r+1}$, include $y_{r+1}$ and exclude $y_{r-k+1}$, i.e., store the new word $y_{r-k+2}{y}_{r-k+2}\dots y_{r+1}$. Then, calculate the new $H_k:=H_k\oplus y_{r+1}\oplus y_{r-k+1}$, read the new letter $x_{r+2}$ and so on.

Theorem 4.

The transformation ${\tau }_k^{*}$ equals ${\tau }_k$, and, hence, the above-described algorithm performs the transformation ${\tau }_k$ in time $O\left(1\right)$ using memory $O\left(k\right)$ bits.

The proof is given in Appendix A.

4. ∞-Distributed Processes

The k-order two-faced process is k distributed. Here, we describe ∞-distributed processes. We call suggested processes as normal two-faced.

Definition 4.

If, for any binary word v, Equation (9) is true, then the process is called normal two-faced, while, if Equation (10) is true, the process is asymptotically normal two-faced.

Now, we describe a family of such processes. Suppose that $m^{*}=m_1,m_2,\dots .$ is a sequence of integers, $m_1<m_2<m_3\dots .$ and $X^1=x_1^1{x}_2^1\dots$, $X^2=x_1^2{x}_2^2\dots$, $X^3=x_1^3{x}_2^3\dots$, $\dots$ are (asymptotically) two-faced processes of order $m_1,m_2,\dots$, correspondingly. Define a process W$=w_1{w}_2$$\dots$ by the following equation:

$$w_i=\left\{\begin{array}{c}{x}_i^{1}i\le m_1,\\ x_i^1\oplus x_i^2{m}_1<i\le m_2,\\ x_i^1\oplus x_i^2\oplus x_i^3{m}_2<i\le m_3,\\ \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \end{array}\right.$$

(17)

and denote it as ${⨁}_{i=1}^{\infty }{X}^i$.

Theorem 5.

Let all $X^i$, $i=1,2,\dots$, be two-faced. Then, ${⨁}_{i=1}^{\infty }{X}^i$ is normal two-faced. If $X^i$, $j=1,2,\dots$ are asymptotically two-faced, then ${⨁}_{i=1}^{\infty }{X}^i$ is asymptotically normal two-faced.

The proof is given in Appendix A. From this and Theorem 2, we can derive the following.

Corollary 1.

If $X=x_1{x}_2\dots$ and $Y=y_1{y}_2\dots$ are stochastic processes and X is normal two-faced, then the process Z $=z_1{z}_2\dots$, $z_1=x_1\oplus y_1$, $z_2=x_2\oplus y_2,\dots$ is normal two-faced.

Note that the entropy of the processes $X^1,X^2,\dots$ can be small; hence, the entropy of the process ${⨁}_{i=1}^{\infty }{X}^i$ can be arbitrary small. On the other hand, the process looks truly random.

5. Experiments

Here, we present some experiments describing the two-faced processes with different parameters. We compared obtained sequences with truly random applying the ${\chi }^2$ test [23]. For this purpose, N-letter sequence $x_1{x}_2\dots x_N$, $N=1000$, were generated, whereas the initial part $x_{-k+1}\dots x_0$ was uniformly distributed. The sequence $x_1{x}_2\dots x_N$ was presented as $x_1{x}_2{x}_k$, $x_{k+1}{x}_{k+2}\dots x_{2k}$ , $\dots$ and the frequency of occurrence of all words from ${\{0,1\}}^k$ was estimated. Then,

$$x^2=\sum _{w\in {\{0,1\}}^k}{(\nu \left(w\right)-(\lfloor N/k\rfloor /2^k))}^2/((\lfloor N/k\rfloor /2^k)$$

was calculated, where $N=1000$, $\nu \left(w\right)$ is the number of occurrences of w in the sequence $x_1{x}_2{x}_k$, $x_{k+1}{x}_{k+2}\dots x_{2k},$$\dots$. (Note that $x^2$ estimates the frequency deviation from the uniform distribution.) Then, $x^2$ was compared with the quantile ${\chi }_{d,0.99}^2$, where $d=2^k-1$; see [23]). If $x^2$$>{\chi }_{d,0.99}^2$, we rejected $H_0$. Table 1 contains results of calculations. (The entropy is equal to $-(p{log}_{2}p+(1-p){log}_2(1-p))$).

Table 1. Two-faced processes testing.

$\mathit{\pi }$	k	Accepted	Entropy (Bits)
0.3	2	10	0.88
0.3	3	10	0.88
0.3	4	10	0.88
0.3	5	10	0.88
0.2	2	9	0.72
0.2	3	9	0.72
0.2	4	10	0.72
0.2	5	10	0.72
0.1	2	9	0.47
0.1	3	9	0.47
0.1	4	9	0.47
0.1	5	9	0.47
0.05	2	10	0.29
0.05	3	8	0.29
0.05	4	5	0.29
0.05	5	4	0.29
0.01	2	1	0.08
0.01	3	4	0.08
0.01	4	1	0.08
0.01	5	0	0.08

Thus, we can see that the two-faced processes can be obtained from low-entropic ones.

6. Conclusions

In this paper, we describe low-entropic processes which mimic truly random ones. In other words, the output is either ∞-distributed or k-distributed for some integer k. In addition, we show how those processes can be directly used in order to construct (or “improve ”) PRNGs.