Research on the Characteristics of Joint Distribution Based on Minimum Entropy

Abstract

This paper focuses on the extreme-value issue of Shannon entropy for joint distributions with specified marginals, a subject of growing interest. It introduces a theorem showing that the coupling with minimal entropy must be essentially order-preserving, whereas the coupling with maximal entropy aligns with independence. This means that the minimum-entropy coupling in a two-dimensional system forms an upper triangular discrete joint distribution by exchanging the rows and columns of the joint distribution matrix. Consequently, entropy is interpreted as a measure of system disorder. This manuscript’s key academic contribution is in clarifying the physical meaning behind optimal-entropy coupling, where a special ordinal relationship is pinpointed and methodically outlined. Furthermore, it offers a computational approach for order-preserving coupling as a practical illustration.

1. Introduction and Statement of the Result

The concept of entropy was introduced in thermodynamic and statistical mechanics as a measure of uncertainty or disorganization in a physical system [1,2]. In 1877, L. Boltzmann [2] gave the probabilistic interpretation of entropy and found the famous formula $S=\kappa logW$. Roughly speaking, entropy is the logarithm of the number of ways in which the physical system can be configured. The second law of thermodynamics says that the entropy of a closed system cannot decrease.

To reveal the physics of information, C. Shannon [3] introduced entropy into communication theory. Let X be a discrete random element with alphabet $\mathcal{X}$ and probability mass $\mathbf{p}=\left\{p\right(x)=\mathbb{P}(X=x):x\in \mathcal{X}\}$, the entropy of X (or $\mathbf{p}$) is defined by

$$H\left(X\right)=H\left(\mathbf{p}\right):=-\sum _{x\in \mathcal{X}}p\left(x\right)logp\left(x\right).$$

(1)

Clearly, $H\left(X\right)$, which is called the Shannon entropy, takes its minimum 0 when X is degenerated and takes its maximum $log\left|\mathcal{X}\right|$ when X is uniformly distributed. In this sense, entropy is a measure of the uncertainty of a random element.

In the theory of information, the definition of entropy is extended to a pair of random variables as follows. Let $(X,Y)$ be a two-dimensional random vector in $\mathcal{X}\times \mathcal{Y}$ with a joint distribution P = $\left\{p\right(x,y):x\in \mathcal{X},y\in \mathcal{Y}\}$; the joint entropy of $(X,Y)$ (or P) is defined by

$$H(X,Y)=H\left(P\right):=-\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}p(x,y)logp(x,y).$$

(2)

Another important concept for $(X,Y)$ is mutual information, which is a measure of the amount of information that one random variable contains about the other. It is defined by

$$I(X,Y):=\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}p(x,y)log{\displaystyle \frac{p(x,y)}{p\left(x\right)p\left(y\right)}},$$

(3)

where $\left\{p\right(x):x\in \mathcal{X}\}$ and $\left\{p\right(y):y\in \mathcal{Y}\}$ are the marginal distributions of X and Y. By definition, one has

$$I(X,Y)=H\left(X\right)+H\left(Y\right)-H(X,Y).$$

(4)

Note that in some settings, the maximum of a variable’s mutual information is called its channel capacity, which plays a key role in information theory through the famous Shannon’s second theorem: the Channel Coding Theorem [3].

For basic concepts and properties in information theory, readers may refer to [4] and the references therein. For entropy and its developments in various topics of pure mathematics, readers may pay attention to the series of talks given by Xiang-Dong Li [5].

By (4), for the given marginals $\left\{p\right(x):x\in \mathcal{X}\}$ and $\left\{p\right(y):y\in \mathcal{Y}\}$, maximizing $I(X,Y)$ and minimizing $H(X,Y)$ are two sides of a single coin. To infer an unknown joint distribution of two random variables with given marginals is an old problem in the area of probabilistic inference. As far as we know, the problem goes back to at least Frechet [6] and Hoeffding [7,8], who studied the question of identifying the extremal joint distribution that maximized (or minimized, respectively) their correlation. For more studies in this area and more applications in pure and applied sciences, readers may refer to [9,10,11,12,13,14], etc.

In this paper, we consider the following setting of the problem described above. For simplicity, suppose $\mathcal{X}=\mathcal{Y}=\{1,2,\dots ,n\},n\ge 2$, and $\mathbf{p}$$=(p_1,p_2,\dots ,p_n)$ and $\mathbf{q}$$=(q_1,q_2,\dots ,q_n)$ be two discrete probability distributions on $\mathcal{X}$. Let X and Y be the random variables in $\mathcal{X}$, with distributions $\mathbf{p}$ and $\mathbf{q}$, respectively; then, we try to seek a minimum-entropy of two-dimensional random vector $(X,Y)$ in $\mathcal{X}\times \mathcal{X}$ with the marginals $\mathbf{p}$ and $\mathbf{q}$.

One strategy to solve the problem mentioned above is to calculate the exact value of the minimum entropy, $H(X,Y)$. Our efforts in this direction will be reported in a following paper [15]: depending on the local optimization lemmas given in Section 2, we successfully obtain an algorithm to calculate the exact value of the minimal joint entropy for any given marginals $\mathbf{p}$ and $\mathbf{q}$. Note that the computation complexity for our algorithm is $\left({\displaystyle \genfrac{}{}{0pt}{}{n^2}{2n}}\right)$.

Another strategy to study the problem is to seek the unknown special structure of a minimum-entropy coupling $(X,Y)$. Clearly, in cases where X and Y are independent, the joint entropy $H(X,Y)$ takes the maximum $H\left(X\right)+H\left(Y\right)$. That is to say, the independent structure (maybe the most disordered one) of $(X,Y)$ determines the maximum entropy, but what special structure in a coupling $(X,Y)$ will determine the minimum entropy of a two-dimensional random system? The main goal of the present paper is to establish such a structure.

Denote by ${\mathcal{P}}_n$ the set of all discrete probability distributions on $\mathcal{X}$$=\{1,$$2,\dots ,n\}$. For each $\mathbf{p}\in {\mathcal{P}}_{\mathbf{n}}$, let $F_{\mathbf{p}}$ be the cumulative distribution function, defined by

$$F_{\mathbf{p}}\left(i\right):=\sum _{k=1}^i{p}_k,1\le i\le n.$$

(5)

Recall that a permutation $\sigma$ is a bijective map from $\mathcal{X}$$=\{1,2,\dots ,n\}$ into itself. For any $\mathbf{p}=(p_1,p_2,\dots ,p_n)\in {\mathcal{P}}_n$, define $\sigma \mathbf{p}:=(p_{\sigma \left(1\right)},p_{\sigma \left(2\right)},\dots ,p_{\sigma \left(n\right)})$. From the Equation of (1), one has

$$H\left(\mathbf{p}\right)=H\left(\sigma \mathbf{p}\right)$$

(6)

which holds for any permutation $\sigma$. For a random variable X with distribution $\mathbf{p}$, the random variable $\sigma X$ has the distribution ${\sigma }^{-1}\mathbf{p}$, where ${\sigma }^{-1}$ is the inverse of $\sigma$.

For any $\mathbf{p},\mathbf{q}\in {\mathcal{P}}_n$, denote by $\mathcal{C}(\mathbf{p},\mathbf{q})$ the set of all joint distributions P with the marginals $\mathbf{p},\mathbf{q}$. For any $P\in \mathcal{C}(\mathbf{p},\mathbf{q})$, suppose $(X,Y)$ is distributed according to P. For any permutation pair $(\sigma ,\pi )$, denote by $P(\sigma ,\pi )$ the joint distribution of $(\sigma X,\pi Y)$. Then,

$$P(\sigma ,\pi )\in \mathcal{C}\left({\sigma }^{-1}\mathbf{p},{\pi }^{-1}\mathbf{q}\right)\mathrm{and}H\left(P(\sigma ,\pi )\right)=H\left(P\right).$$

(7)

For any $1\le k,l\le n$, denote by $E(k,l)={\left(e_{i,j}\right)}_{n\times n}$ the matrix such that $e_{i,i}=1$;$i\ne k,l$; $e_{k,l}=e_{l,k}=1$; and $e_{i,j}=0$. Otherwise, let $\mathcal{E}$$:=\left\{E\right(k,l):1\le k,l\le n\}$. For any n-th order probability matrix P, let $P^{\prime }=E(k,l)·P$ (resp. $P·E(k,l)$); then, $P^{\prime }$ is the matrix obtained from P by exchanging the positions of its k-th and l-th rows (resp. columns). Furthermore, for any $P\in \mathcal{C}(\mathbf{p},\mathbf{q})$ and any finite sequences $\{E_k\in \mathcal{E}:1\le k\le s\}$ and $\{E_l^{\prime }\in \mathcal{E}:1\le l\le t\}$, there exists a unique permutation pair $(\sigma ,\pi )$, such that

$$P(\sigma ,\pi )=E_1·E_2·\dots ·E_s·P·E_t^{\prime }·E_{t-1}^{\prime }·\dots ·E_2^{\prime }·E_1^{\prime }.$$

(8)

Definition 1. 

For any $\mathbf{p},\mathbf{q}\in {\mathcal{P}}_n$, suppose $(X,Y)$ is distributed according to $P\in \mathcal{C}(\mathbf{p},\mathbf{q})$. We note the following:

1. 

$(X,Y)$, or P, is order-preserving, if $\mathbb{P}(X\le Y)=1$;

2. 

$(X,Y)$, or P, is essentially order-preserving, if for some permutation pair $(\sigma ,\pi )$,

$$\mathbb{P}(\sigma X\le \pi Y)=1.$$

$(X,Y)$ being order-preserving is also called “X being stochastically dominated by Y” in probability theory. Clearly, it holds if and any if any $1\le j<i\le n$, $\mathbb{P}(X=i,Y=j)=p_{i,j}=0$, i.e., the joint distribution $P=\left(p_{i,j}\right)$ is upper triangular. For an essentially order-preserving $P\in \mathcal{C}(\mathbf{p},\mathbf{q})$, for the permutation pair $(\sigma ,\pi )$ given in Definition 1, $P(\sigma ,\pi )$ is upper triangular.

For any $\mathbf{p},\mathbf{q}\in {\mathcal{P}}_{\mathbf{n}}$, by Strassen’s Theorem [16] on stochastic domination, there exists a coupling $P\in \mathcal{C}(\mathbf{p},\mathbf{q})$ such that P is upper triangular if and only if $F_{\mathbf{q}}\le F_{\mathbf{p}}$, where $F_{\mathbf{p}}$ and $F_{\mathbf{q}}$ are defined in (5). Note that in the literature on information theory (see [11,17] etc.), people usually say that “$\mathbf{q}$ is majorized by $\mathbf{p}$” when $F_{\mathbf{q}}\le F_{\mathbf{p}}$.

Now, we turn to the following optimization problem:

$$\tilde{P}:H\left(\tilde{P}\right)=\underset{P\in \mathcal{C}(\mathbf{p},\mathbf{q})}{inf}H\left(P\right).$$

(9)

Note that $\mathcal{C}(\mathbf{p},\mathbf{q})$ forms a compact subset of ${\mathbb{R}}^{n^2}$; the existence of such a $\tilde{P}$ follows from the continuity of the entropy function H. Note that in this paper, we also call $\tilde{P}$ the minimum-entropy coupling.

What key structural characteristics of a coupling (X,Y) govern the minimum entropy in the two-dimensional stochastic system, as formulated in optimization problem (9)? Specifically, what mathematical properties must the joint distribution (X,Y) satisfy to achieve the global minimization of the joint entropy? Is there an intrinsic connection between such optimal coupling structures and the order-preserving properties of the variables, as suggested by entropy minimization principles? We state our main result as follows.

Theorem 1. 

Suppose $n\ge 2$ and $\mathbf{p},\mathbf{q}$$\in {\mathcal{P}}_n$. If $\tilde{P}\in \mathcal{C}(\mathbf{p},\mathbf{q})$ solves the optimization problem (9), and $(X,Y)$ is distributed according to $\tilde{P}$, then $(X,Y)$, or $\tilde{P}$, is essentially order-preserving.

Theorem 1 shows that the coupling with the minimal entropy must be essentially order-preserving, whereas the coupling with the maximal entropy aligns with independence. This means that the minimum-entropy coupling in a two-dimensional discrete system must be an upper triangular discrete joint distribution which can be formed by exchanging the rows and columns of the joint distribution matrix based on Theorem 3. Consequently, entropy is interpreted as a measure of system disorder.

Another structure of the minimum-entropy coupling is discovered in paper [15]: a tree structure in graph theory. Based on this structure, we develop an algorithm to obtain the exact value of the minimal entropy.

The rest of the paper is arranged as follows. In Section 2.1, we first develop the following local optimization lemmas: Lemma 2 and Lemma 3. Then, for any $P\in \mathcal{C}(\mathbf{p},\mathbf{q})$, by using the local optimization lemmas, we construct a $P^{\prime }\in \mathcal{C}(\mathbf{p},\mathbf{q})$ such that $H\left(P\right)\ge H\left(P^{\prime }\right)$. In Section 2.2, by Lemma 2, developed in Section 2.1, we prove Theorem 1. In Section 3, for $n=6$ for P, the independent coupling of $\mathbf{p},\mathbf{q}\in {\mathcal{P}}_{\mathbf{6}}$ is given as an example for Theorem 3, and we optimise P to an upper triangular $P^{\prime }$.

2. Local Optimization and Proof of Result

2.1. Local Optimization Lemmas

Suppose $A={\left(a_{i,j}\right)}_{n\times n}$ is a nonnegative matrix (i.e., all its entries are nonnegative) such that $C:={\sum }_{i=1}^n{\sum }_{j=1}^n{a}_{i,j}>0$. We generalize the definition of entropy for nonnegative matrix A as

$$H\left(A\right)=-h\left(A\right):=-\sum _{i=1}^n\sum _{j=1}^n{a}_{i,j}log{a}_{i,j}.$$

(10)

Let $P=C^{-1}A$, a probability matrix, then

$$H\left(A\right)=CH\left(P\right)-ClogC.$$

(11)

For any $c>0$, define $h_c\left(x\right)=xlogx+(c-x)log(c-x)$ and $x\in [0,c]$. Before the local optimization lemmas, we give out the following simple property for function $h_c$ without proof.

Lemma 1. 

For any closed interval $[a,b]\subseteq [0,c]$, $max\{h_c\left(x\right):x\in [a,b]\}=h_c\left(a\right)\vee h_c\left(b\right)$$=h_c\left(a\right)I_{\{a\le c-b\}}+h_c\left(b\right)I_{\{a>c-b\}}.$

Lemma 2. 

For any second-order nonnegative matrix $A={\left(a_{i,j}\right)}_{2\times 2}$, suppose that $a_{1,1}\vee a_{2,2}\ge a_{1,2}\vee a_{2,1}$ and denote $b=a_{1,2}\wedge a_{2,1}$. Let $A^{\prime }={\left(a_{i,j}^{\prime }\right)}_{2\times 2}$ such that $a_{i,i}^{\prime }=a_{i,i}+b,i=1,2$, $a_{i,j}^{\prime }=a_{i,j}-b$, and $i\ne j.$ Then, $H\left(A\right)\ge H\left(A^{\prime }\right)$. Furthermore, if $b>0$, then $H\left(A\right)>H\left(A^{\prime }\right)$.

Proof. 

Without loss of generality, assume that $a_{1,1}\ge a_{2,1}\ge a_{1,2}$. Note that in this case, one has $b=a_{1,2}$ and $a_{1,2}^{\prime }=0$. For any $x\in [0,b]$, define

$$A\left(x\right)=\left(\begin{array}{cc}{a}_{1,1}+x\hfill & a_{1,2}-x\hfill \\ a_{2,1}-x\hfill & a_{2,2}+x\hfill \end{array}\right).$$

Let $c_1=a_{1,1}+a_{2,1}$ and $c_2=a_{1,2}+a_{2,2}$; then, $h\left(A\left(x\right)\right)=h_{c_1}(a_{1,1}+x)+h_{c_2}(a_{1,2}-x)$.

By Lemma 1, and the assumption given above, $max\{h_{c_1}(a_{1,1}+x):x\in [0,b]\}=h_{c_1}(a_{1,1}+b)$ and $max\{h_{c_2}(a_{1,2}-x):x\in [0,b]\}=h_{c_2}\left(0\right).$ Thus, $max\left\{h\right(A\left(x\right)):x\in [0,b\left]\right\}=h\left(A\right(b\left)\right)$ and $h\left(A\right)=h\left(A\left(0\right)\right)\le h\left(A\left(b\right)\right)=h\left(A^{\prime }\right)$; then, by definition $H\left(A\right)\ge H\left(A^{\prime }\right)$.

If $b=a_{1,2}>0$, then $a_{1,1}>a_{2,1}-a_{1,2}$. By Lemma 1, $h_{c_1}\left(a_{1,1}\right)<h_{c_1}(a_{1,1}+a_{1,2})$; this implies that $h\left(A\right)<h\left(A^{\prime }\right)$ and $H\left(A\right)>H\left(A^{\prime }\right)$. □

Lemma 3. 

For any second-order nonnegative matrix $A={\left(a_{i,j}\right)}_{2\times 2}$, suppose that $a_{1,1}+a_{1,2}\ge a_{2,1}+a_{2,2}$, $a_{1,1}+a_{2,1}\ge a_{1,2}+a_{2,2}$, and $a_{1,1}+a_{1,2}\ge a_{1,1}+a_{2,1}$. Let $b=a_{1,2}\wedge a_{2,1}$ and define $A^{\prime }$ as in Lemma 2; then, $H\left(A\right)\ge H\left(A^{\prime }\right)$.

Proof. 

By Relation (11), without loss of generality, assume A to be a probability matrix. Rewrite the probability matrix by $P={\left(p_{i,j}\right)}_{2\times 2}$, and write $\mathbf{p}=(p_1,p_2)=(p,1-p)$ and $\mathbf{q}=(q_1,q_2)=(q,1-q)$ as its marginals, i.e., $p_{1,1}+p_{1,2}=p$ and $p_{2,1}+p_{2,2}=1-p$, and $p_{1,1}+p_{2,1}=q$ and $p_{1,2}+p_{2,2}=1-q$. By the conditions of the lemma, one has $p\ge 1-p$, $q\ge 1-q$ and $p\ge q\ge {\displaystyle \frac{1}{2}}$.

For any $x\in [0,1-p]$, let

$$P\left(x\right)=\left(\begin{array}{cc}q-x\hfill & (p-q)+x\hfill \\ x\hfill & (1-p)-x\hfill \end{array}\right).$$

Clearly, $P\left(x\right)$ is a joint probability distribution matrix with marginals $\mathbf{p},\mathbf{q}$, and $P\left(p_{2,1}\right)=P$ and $P\left(0\right)=P^{\prime }$. Particularly, when x runs over the interval $[0,1-p]$, $P\left(x\right)$ runs over $\mathcal{C}(\mathbf{p},\mathbf{q})$.

Now, $h\left(P\left(x\right)\right)=h_p(q-x)+h_{1-p}\left(x\right)$. It follows immediately from Lemma 1 that $h_{1-p}\left(x\right)$ and $h_p(q-x)$ take their maximums simultaneously when $x=0$; then, $h\left(P\right)=h\left(P\left(p_{2,1}\right)\right)\le h\left(P\left(0\right)\right)=h\left(P^{\prime }\right)$ and $H\left(P\right)\ge H\left(P^{\prime }\right)$. □

From the statements of Lemmas 2 and 3, for any second-order nonnegative matrix A, one may easily construct the nonnegative matrix $A^{\prime }$, which possesses the same row and column summations as A, such that $H\left(A\right)\ge H\left(A^{\prime }\right)$.

As a consequence of Lemma 3, the optimization problem (9) for $n=2$ is solved as follows.

Corollary 1. 

For $\mathbf{p},\mathbf{q}\in {\mathcal{P}}_2$ such that $\mathbf{p}=(p,1-p)$, $\mathbf{q}=(q,1-q)$, and $p\ge q\ge {\displaystyle \frac{1}{2}}$, let

$$\tilde{P}=\left(\begin{array}{cc}q\hfill & p-q\hfill \\ 0\hfill & 1-p\hfill \end{array}\right),$$

then

$$H\left(\tilde{P}\right)=\underset{P\in \mathcal{C}(\mathbf{p},\mathbf{q})}{inf}H\left(P\right)=-[qlogq+(1-p)log(1-p)+(p-q)log(p-q)].$$

Now, as a direct consequence of Lemmas 2 and 3, we obtain the following local optimization theorem.

Theorem 2. 

Suppose $\mathbf{p},\mathbf{q}\in {\mathcal{P}}_n$, $n\ge 2$, and $P\in \mathcal{C}(\mathbf{p},\mathbf{q})$. For any $1\le i<k\le n$ and $1\le j<l\le n$, let

$$A=\left(\begin{array}{cc}{p}_{i,j}\hfill & p_{i,l}\hfill \\ p_{k,j}\hfill & p_{k,l}\hfill \end{array}\right)$$

(12)

be the second-order submatrix of P. Let $A^{\prime }$ be given as in Lemma 2 or Lemma 3 and let $P^{\prime }$ be the matrix obtained from P by $A^{\prime }$ taking the place of A. Then, $P^{\prime }\in \mathcal{C}(\mathbf{p},\mathbf{q})$ and $H\left(P\right)\ge H\left(P^{\prime }\right)$.

2.2. Proof of Theorem 1

In this subsection, we give proof to Theorem 1. Actually, the strategy of the proof is to use the local optimization Lemma 2 repeatedly. First of all, we have the following lemma.

Lemma 4. 

Let $A={\left(a_{i,j}\right)}_{n\times n},n\ge 2$ be an n-th-order nonnegative matrix. Suppose that

$$a_{1,1}=\underset{1\le i,j\le n}{max}{a}_{i,j},\sum _{j=1}^n{a}_{1,j}\ge \sum _{i=1}^n{a}_{i,1}$$

(13)

$$\left(\mathrm{resp}.a_{n,n}=\underset{1\le i,j\le n}{max}{a}_{i,j},\sum _{j=1}^n{a}_{n,j}\le \sum _{i=1}^n{a}_{i,n}\right).$$

Then, by using the local optimization procedure developed in Lemma 2 and Theorem 2 at most $2(n-1)$ times, finally transform A to $A^{\prime }={\left(a_{i,j}^{\prime }\right)}_{n\times n}$ such that $H\left(A\right)\ge H\left(A^{\prime }\right)$,

$$\sum _{i=1}^n{a}_{i,j}=\sum _{i=1}^n{a}_{i,j}^{\prime },\sum _{j=1}^n{a}_{i,j}=\sum _{j=1}^n{a}_{i,j}^{\prime },1\le i,j\le n,\mathrm{and}$$

(14)

$$a_{1,1}^{\prime }=\sum _{i=1}^n{a}_{i,1},a_{i,1}^{\prime }=0,2\le i\le n$$

(15)

$$\left(\mathrm{resp}.a_{n,n}^{\prime }=\sum _{j=1}^n{a}_{n,j},a_{n,j}^{\prime }=0,1\le j\le n-1\right).$$

Furthermore, $H\left(A\right)=H\left(A^{\prime }\right)$ if and only if $a_{1,1}^{\prime }=a_{1,1}$$\left(\mathrm{resp}.a_{n,n}^{\prime }=a_{n,n}\right).$

Proof. 

Let $I=\{i:a_{i,1}>0,2\le i\le n\}$ and $J=\{j:a_{1,j}>0,2\le j\le n\}$, and denote by $\left|I\right|$ and $\left|J\right|$ the cardinalities of I and J. Without loss of generality, assume that $i_0\in I,j_0\in J$ satisfies

$$a_{i_0,1}=\underset{i\in I}{min}{a}_{i,1}\wedge \underset{j\in J}{min}{a}_{1,j},a_{1,j_0}=\underset{j\in J}{min}{a}_{1,j}.$$

Write $b=a_{i_0,1}$, and renew A by changing the second-order submatrix

$$\left(\begin{array}{cc}{a}_{1,1}\hfill & a_{1,j_0}\hfill \\ a_{i_0,1}\hfill & a_{i_0,j_0}\hfill \end{array}\right)\mathrm{to}\left(\begin{array}{cc}{a}_{1,1}+b\hfill & a_{1,j_0}-b\hfill \\ a_{i_0,1}-b\hfill & a_{i_0,j_0}+b\hfill \end{array}\right).$$

Then, the renewed A still satisfies the conditions of Lemma 2, and then, by this lemma, its entropy $H\left(A\right)$ is decreased strictly. Note that after using the local optimization procedure once, for the renewed matrix A, the number $\left|I\right|+\left|J\right|$ decreases by 1.

Repeat the above procedure until $I=\varnothing$ and write $A^{\prime }$ as the final renewing of A; thus, $A^{\prime }$ is obtained as required. □

In the situation given as in the statement of Lemma 4, we denote $L_n$ as the corresponding composite optimization procedure, and write $A^{\prime }=L_n\left(A\right)$.

Now, we finish the proof of Theorem 1 by proving the following theorem.

Theorem 3. 

For any $\mathbf{p},\mathbf{q}\in {\mathcal{P}}_n,n\ge 2$, $P\in \mathcal{C}(\mathbf{p},\mathbf{q})$, there exists a permutation pair $(\overline{\sigma },\overline{\pi })$ and an upper triangular $P^{\prime }\in \mathcal{C}({\overline{\sigma }}^{-1}\mathbf{p},{\overline{\pi }}^{-1}\mathbf{q})$ such that $H\left(P\right)\ge H\left(P^{\prime }\right)$. Furthermore, cases where $P(\sigma ,\pi )$ is not upper triangular for any permutation pair $(\sigma ,\pi )$, $H\left(P\right)>H\left(P^{\prime }\right)$.

Proof. 

For $n=2$, Theorem 3 follows from Lemma 2 immediately. For general $n>2$ and $P={\left(p_{i,j}\right)}_{n\times n}\in \mathcal{C}(\mathbf{p},\mathbf{q})$, suppose ${\displaystyle p_{i_0,j_0}=\underset{1\le i,j\le n}{max}{p}_{i,j}}$. In the case of

$$\sum _{j=1}^n{p}_{i_0,j}\ge \sum _{i=1}^n{p}_{i,j_0}\left(\mathrm{resp}.\sum _{j=1}^n{p}_{i_0,j}\le \sum _{i=1}^n{p}_{i,j_0}\right),$$

let $A=E(1,i_0)·P·E(1,j_0)$ (resp. $E(n,i_0)·P·E(n,j_0)$), where $E(1,i_0)$, $E(1,j_0)$, $E(n,i_0)$, and $E(n,j_0)\in \mathcal{E}$. By (8), for some given permutation pair $({\sigma }_1,{\pi }_1)$, $A=P({\sigma }_1,{\pi }_1)$ and A satisfies Condition (13). By Lemma 4, Equations (14) and (15) hold for $A^{\prime }=L_n\left(A\right)$.

Note that by (14), A and $A^{\prime }$ are all the probability matrixes in $\mathcal{C}({\sigma }_1^{-1}\mathbf{p},{\pi }_1^{-1}\mathbf{q})$, and $H\left(P\right)=H\left(A\right)\ge H\left(A^{\prime }\right)$.

Now, $A^{\prime }$ has the form

$$A^{\prime }=\left(\begin{array}{ccccc}{a}_{1,1}^{\prime }\hfill & a_{1,2}^{\prime }\hfill & \cdots \hfill & a_{1,n-1}^{\prime }\hfill & a_{1,n}^{\prime }\hfill \\ 0\hfill & a_{2,2}^{\prime }\hfill & \cdots \hfill & a_{2,n-1}^{\prime }\hfill & a_{2,n}^{\prime }\hfill \\ ⋮\hfill & ⋮\hfill & \ddots \hfill & ⋮\hfill & ⋮\hfill \\ 0\hfill & a_{n-1,2}^{\prime }\hfill & \cdots \hfill & a_{n-1,n-1}^{\prime }\hfill & a_{n-1,n}^{\prime }\hfill \\ 0\hfill & a_{n,2}^{\prime }\hfill & \cdots \hfill & a_{n,n-1}^{\prime }\hfill & a_{n,n}^{\prime }\hfill \end{array}\right)$$

(16)

$$\left(\mathrm{resp}.\left(\begin{array}{ccccc}{a}_{1,1}^{\prime }\hfill & a_{1,2}^{\prime }\hfill & \cdots \hfill & a_{1,n-1}^{\prime }\hfill & a_{1,n}^{\prime }\hfill \\ a_{2,1}^{\prime }\hfill & a_{2,2}^{\prime }\hfill & \cdots \hfill & a_{2,n-1}^{\prime }\hfill & a_{2,n}^{\prime }\hfill \\ ⋮\hfill & ⋮\hfill & \ddots \hfill & ⋮\hfill & ⋮\hfill \\ a_{n-1,1}^{\prime }\hfill & a_{n-1,2}^{\prime }\hfill & \cdots \hfill & a_{n-1,n-1}^{\prime }\hfill & a_{n-1,n}^{\prime }\hfill \\ 0\hfill & 0\hfill & \cdots \hfill & 0\hfill & a_{n,n}^{\prime }\hfill \end{array}\right)\right).$$

Consider the following $(n-1)$-th-order submatrix: $\overline{A}={\left({\overline{a}}_{i,j}\right)}_{(n-1)\times (n-1)}$ of $A^{\prime }$ with ${\overline{a}}_{i,j}=a_{i+1,j+1}^{\prime }$ (resp. $a_{i,j}^{\prime }$) for $1\le i,j\le n-1$,

$$\overline{A}=\left(\begin{array}{cccc}{a}_{2,2}^{\prime }\hfill & \cdots \hfill & a_{2,n-1}^{\prime }\hfill & a_{2,n}^{\prime }\hfill \\ ⋮\hfill & \ddots \hfill & ⋮\hfill & ⋮\hfill \\ a_{n-1,2}^{\prime }\hfill & \cdots \hfill & a_{n-1,n-1}^{\prime }\hfill & a_{n-1,n}^{\prime }\hfill \\ a_{n,2}^{\prime }\hfill & \cdots \hfill & a_{n,n-1}^{\prime }\hfill & a_{n,n}^{\prime }\hfill \end{array}\right)$$

$$\left(\mathrm{resp}.\left(\begin{array}{cccc}{a}_{1,1}^{\prime }\hfill & a_{1,2}^{\prime }\hfill & \cdots \hfill & a_{1,n-1}^{\prime }\hfill \\ a_{2,1}^{\prime }\hfill & a_{2,2}^{\prime }\hfill & \cdots \hfill & a_{2,n-1}^{\prime }\hfill \\ ⋮\hfill & ⋮\hfill & \ddots \hfill & ⋮\hfill \\ a_{n-1,1}^{\prime }\hfill & a_{n-1,2}^{\prime }\hfill & \cdots \hfill & a_{n-1,n-1}^{\prime }\hfill \end{array}\right)\right),$$

suppose $a_{i_1,j_1}^{\prime }$ to be the entry of $\overline{A}$ such that ${\displaystyle a_{i_1,j_1}^{\prime }=\underset{1\le i,j\le n-1}{max}{\overline{a}}_{i,j}=\underset{2\le i,j\le n}{max}{a}_{i,j}^{\prime }}$ (resp. ${\displaystyle \underset{1\le i,j\le n-1}{max}{a}_{i,j}^{\prime }}$).

For simplicity, we only treat the first case of $\overline{A}$ (the resp. case in the brackets can be treated similarly); of course, in this case, one has $2\le i_1,j_1\le n.$

In the subcase when

$$\sum _{i=2}^n{a}_{i,j_1}^{\prime }\le \sum _{j=2}^n{a}_{i_1,j}^{\prime }\left(\mathrm{resp}.\sum _{i=2}^n{a}_{i,j_1}^{\prime }\ge \sum _{j=2}^n{a}_{i_1,j}^{\prime }\right),$$

let $A^{″}={\left(a_{i,j}^{″}\right)}_{n\times n}=E(2,i_1)·A^{\prime }·E(2,j_1)$ (resp. $E(n,i_1)·A^{\prime }·E(n,j_1)$). Clearly, by (8), for some given permutation pair $({\sigma }_2,{\pi }_2)$, $A^{″}=A^{\prime }({\sigma }_2,{\pi }_2)\in \mathcal{C}\left({\left({\sigma }_1{\sigma }_2\right)}^{-1}\mathbf{p},{\left({\pi }_1{\pi }_2\right)}^{-1}\mathbf{q}\right)$, and $H\left(P\right)\ge H\left(A^{\prime }\right)=H\left(A^{″}\right).$ Here, we note the following:

• $A^{″}$ has the same form of $A^{\prime }$ as given in Equation (16), i.e., all entries except for $a_{1,1}^{″}=a_{1,1}^{\prime }$ in the first column of $A^{″}$ are zero. In fact, by transforming P to $A^{″}$, we finished the first step of upper-triangulization;
• $\overline{\overline{A}}:=\left(a_{i,j}^{″}:2\le i,j\le n\right)$, as the $(n-1)$-th-order submatrix of $A^{″}$ satisfies the condition of Lemma 4 for $n-1$.

At this moment, we have finished the first step of the optimization of P by transforming P to $A^{″}$. In fact, we are standing at the position to begin the second step by using Lemma 4 on $\overline{\overline{A}}$ for $n-1$.

Repeat the above procedure for $n-1$, $n-2$, ⋯, and finally for 2; $A^{″}$ is then transformed to $A^{⁗}$, $A^{\left(6\right)}$, and finally to $P^{\prime }=A^{\left(2\right(n-1\left)\right)}$. Additionally, certain permutation pair sequences $\{({\sigma }_k,{\pi }_k):1\le k\le n\}$ further exist, such that,

$$A^{\left(2k\right)}\in \mathcal{C}({({\sigma }_1·\dots ·{\sigma }_{k+1})}^{-1}\mathbf{p},{({\pi }_1·\dots ·{\pi }_{k+1})}^{-1}\mathbf{q}),\forall 1\le k\le n-1,$$

and

$$H\left(P\right)=H\left(A\right)\ge H\left(A^{″}\right)\ge H\left(A^{⁗}\right)\ge \dots \ge H\left(A^{\left(2\right(n-1\left)\right)}\right)=H\left(P^{\prime }\right).$$

Now, let $\overline{\sigma }:={\sigma }_1·\dots ·{\sigma }_n$, $\overline{\pi }:={\pi }_1·\dots ·{\pi }_n$. Then, $P^{\prime }\in \mathcal{C}\left({\overline{\sigma }}^{-1}\mathbf{p},{\overline{\pi }}^{-1}\mathbf{q}\right)$, and by our setting, $P^{\prime }$ is upper triangular.

If $P(\sigma ,\pi )$ is not upper triangular for any permutation pair $(\sigma ,\pi )$, then at least one optimization step carried out above is concrete, i.e., when Lemma 2 is used, the corresponding b is strictly positive. Then, by Lemma 2 and the followed Theorem 2, $H\left(P\right)>H\left(P^{\prime }\right)$. □

3. An Example

Theorem 1 shows that the coupling with the minimal entropy must be essentially order-preserving. How can one construct such entropy-equalizing order-preserving coupling structures? In this section, we offer a computational approach for an order-preserving coupling as a practical illustration. Without loss of generality, let us consider the following example: Let $n=6$, $\mathbf{p}=(0.3,0.15,0.08,0.4,$ $0.03,0.04\left)\right)$, $\mathbf{q}=(0.18,0.15,0.44,0.02,0.03,0.18)$, and $P\in \mathcal{C}(\mathbf{p},\mathbf{q})$ be the independent coupling of $\mathbf{p},\mathbf{q}$:

$$P=\left(\begin{array}{cccccc}0.054\hfill & 0.045\hfill & 0.132\hfill & 0.006\hfill & 0.009\hfill & 0.054\hfill \\ 0.027\hfill & 0.0225\hfill & 0.066\hfill & 0.003\hfill & 0.0045\hfill & 0.027\hfill \\ 0.0144\hfill & 0.012\hfill & 0.0352\hfill & 0.0016\hfill & 0.0024\hfill & 0.0144\hfill \\ 0.072\hfill & 0.06\hfill & 0.176\hfill & 0.008\hfill & 0.012\hfill & 0.072\hfill \\ 0.0054\hfill & 0.0045\hfill & 0.0132\hfill & 0.0006\hfill & 0.0009\hfill & 0.0054\hfill \\ 0.0072\hfill & 0.006\hfill & 0.0176\hfill & 0.0008\hfill & 0.0012\hfill & 0.0072\hfill \end{array}\right).$$

Now, we begin to optimise P to a upper triangular $P^{\prime }$, as follows.

First, for the largest entry of P is $p_{4,3}=0.176$ and $q_3=0.44>p_4=0.4$, by exchanging the positions of the 4th and 6th rows and then the positions of the 3rd and 6th columns of P, we obtain

$$A=\left(\begin{array}{cccccc}0.054\hfill & 0.045\hfill & 0.054\hfill & 0.006\hfill & 0.009\hfill & 0.132\hfill \\ 0.027\hfill & 0.0225\hfill & 0.027\hfill & 0.003\hfill & 0.0045\hfill & 0.066\hfill \\ 0.0144\hfill & 0.012\hfill & 0.0144\hfill & 0.0016\hfill & 0.0024\hfill & 0.0352\hfill \\ 0.0072\hfill & 0.006\hfill & 0.0072\hfill & 0.0008\hfill & 0.0012\hfill & 0.0176\hfill \\ 0.0054\hfill & 0.0045\hfill & 0.0054\hfill & 0.0006\hfill & 0.0009\hfill & 0.0132\hfill \\ 0.072\hfill & 0.06\hfill & 0.072\hfill & 0.008\hfill & 0.012\hfill & 0.176\hfill \end{array}\right)$$

By using Lemma 2 on submatrixes

$$\left(\begin{array}{cc}{a}_{k,k}\hfill & a_{k,6}\hfill \\ a_{6,k}\hfill & a_{6,6}\hfill \end{array}\right)\mathrm{for}k=1,2,\dots ,5\mathrm{in}\mathrm{turn},$$

we renew and optimize A to

$$A=\left(\begin{array}{cccccc}0.126\hfill & 0.045\hfill & 0.054\hfill & 0.006\hfill & 0.009\hfill & 0.06\hfill \\ 0.027\hfill & 0.0825\hfill & 0.027\hfill & 0.003\hfill & 0.0045\hfill & 0.006\hfill \\ 0.0144\hfill & 0.012\hfill & 0.0496\hfill & 0.0016\hfill & 0.0024\hfill & 0\hfill \\ 0.0072\hfill & 0.006\hfill & 0.0072\hfill & 0.0088\hfill & 0.0012\hfill & 0.0096\hfill \\ 0.0054\hfill & 0.0045\hfill & 0.0054\hfill & 0.0006\hfill & 0.0129\hfill & 0.0012\hfill \\ 0\hfill & 0\hfill & 0.0368\hfill & 0\hfill & 0\hfill & 0.3632\hfill \end{array}\right);$$

then, using Lemma 2 on submatrix

$$\left(\begin{array}{cc}{a}_{1,3}\hfill & a_{1,6}\hfill \\ a_{6,3}\hfill & a_{6,6}\hfill \end{array}\right)=\left(\begin{array}{cc}0.054\hfill & 0.06\hfill \\ 0.0368\hfill & 0.3632\hfill \end{array}\right),$$

we optimize A to the following

$$A^{\prime }=\left(\begin{array}{cccccc}0.126\hfill & 0.045\hfill & 0.0908\hfill & 0.006\hfill & 0.009\hfill & 0.0232\hfill \\ 0.027\hfill & 0.0825\hfill & 0.027\hfill & 0.003\hfill & 0.0045\hfill & 0.006\hfill \\ 0.0144\hfill & 0.012\hfill & 0.0496\hfill & 0.0016\hfill & 0.0024\hfill & 0\hfill \\ 0.0072\hfill & 0.006\hfill & 0.0072\hfill & 0.0088\hfill & 0.0012\hfill & 0.0096\hfill \\ 0.0054\hfill & 0.0045\hfill & 0.0054\hfill & 0.0006\hfill & 0.0129\hfill & 0.0012\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.4\hfill \end{array}\right).$$

In this situation, we denote

$$\overline{A}=\left(\begin{array}{ccccc}0.126\hfill & 0.045\hfill & 0.0908\hfill & 0.006\hfill & 0.009\hfill \\ 0.027\hfill & 0.0825\hfill & 0.027\hfill & 0.003\hfill & 0.0045\hfill \\ 0.0144\hfill & 0.012\hfill & 0.0496\hfill & 0.0016\hfill & 0.0024\hfill \\ 0.0072\hfill & 0.006\hfill & 0.0072\hfill & 0.0088\hfill & 0.0012\hfill \\ 0.0054\hfill & 0.0045\hfill & 0.0054\hfill & 0.0006\hfill & 0.0129\hfill \end{array}\right),$$

since ${\overline{a}}_{1,1}=0.126$ is the largest entry and

$$\sum _{j=1}^5{\overline{a}}_{1,j}=0.2768>\sum _{i=1}^5{\overline{a}}_{i,1}=0.18,$$

we take $A^{″}=A^{\prime }$, $\overline{\overline{A}}=\overline{A}$, and $\overline{\overline{A}}$ satisfies the conditions of Lemma 4 for $n=5$.

Second, by Lemma 4, $\overline{\overline{A}}$ is optimised to the following

$$\left(\begin{array}{ccccc}0.18\hfill & 0.0036\hfill & 0.0782\hfill & 0.006\hfill & 0.009\hfill \\ 0\hfill & 0.1095\hfill & 0.027\hfill & 0.003\hfill & 0.0045\hfill \\ 0\hfill & 0.0264\hfill & 0.0496\hfill & 0.0016\hfill & 0.0024\hfill \\ 0\hfill & 0.006\hfill & 0.0144\hfill & 0.0088\hfill & 0.0012\hfill \\ 0\hfill & 0.0045\hfill & 0.0108\hfill & 0.0006\hfill & 0.0129\hfill \end{array}\right).$$

Noticing that the largest entry of matrix

$$\left(\begin{array}{cccc}0.1095\hfill & 0.027\hfill & 0.003\hfill & 0.0045\hfill \\ 0.0264\hfill & 0.0496\hfill & 0.0016\hfill & 0.0024\hfill \\ 0.006\hfill & 0.0144\hfill & 0.0088\hfill & 0.0012\hfill \\ 0.0045\hfill & 0.0108\hfill & 0.0006\hfill & 0.0129\hfill \end{array}\right)$$

is $0.1095$, and $0.1095+0.0264+0.006+0.0045=0.1464>$$0.1095+0.027+0.003+0.0045=0.144$, we optimise $A^{″}$ to

$$A^{⁗}=\left(\begin{array}{cccccc}0.18\hfill & 0.009\hfill & 0.0782\hfill & 0.006\hfill & 0.0036\hfill & 0.0232\hfill \\ 0\hfill & 0.0129\hfill & 0.0108\hfill & 0.0006\hfill & 0.0045\hfill & 0.0012\hfill \\ 0\hfill & 0.0024\hfill & 0.0496\hfill & 0.0016\hfill & 0.0264\hfill & 0\hfill \\ 0\hfill & 0.0012\hfill & 0.0144\hfill & 0.0088\hfill & 0.006\hfill & 0.0096\hfill \\ 0\hfill & 0.0045\hfill & 0.027\hfill & 0.003\hfill & 0.1095\hfill & 0.006\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.4\hfill \end{array}\right).$$

Finally, by similar procedures, we optimize $A^{⁗}$ to $A^{\left(6\right)}$, to $A^{\left(8\right)}$, and then to $P^{\prime }=A^{\left(10\right)}$ as follows:

$$A^{\left(6\right)}=\left(\begin{array}{cccccc}0.18\hfill & 0.009\hfill & 0.006\hfill & 0.0782\hfill & 0.0036\hfill & 0.0232\hfill \\ 0\hfill & 0.0174\hfill & 0.0006\hfill & 0.0108\hfill & 0\hfill & 0.0012\hfill \\ 0\hfill & 0.0012\hfill & 0.0118\hfill & 0.0174\hfill & 0\hfill & 0.0096\hfill \\ 0\hfill & 0.0024\hfill & 0.0016\hfill & 0.0736\hfill & 0.0024\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.144\hfill & 0.006\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.4\hfill \end{array}\right);$$

$$A^{\left(8\right)}=\left(\begin{array}{cccccc}0.18\hfill & 0.006\hfill & 0.009\hfill & 0.0782\hfill & 0.0036\hfill & 0.0232\hfill \\ 0\hfill & 0.0118\hfill & 0.0036\hfill & 0.015\hfill & 0\hfill & 0.0096\hfill \\ 0\hfill & 0.0022\hfill & 0.0174\hfill & 0.0092\hfill & 0\hfill & 0.0012\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0.0776\hfill & 0.0024\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.144\hfill & 0.006\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.4\hfill \end{array}\right);$$

$$P^{\prime }=A^{\left(10\right)}=\left(\begin{array}{cccccc}0.18\hfill & 0.006\hfill & 0.009\hfill & 0.0782\hfill & 0.0036\hfill & 0.0232\hfill \\ 0\hfill & 0.014\hfill & 0.0014\hfill & 0.015\hfill & 0\hfill & 0.0096\hfill \\ 0\hfill & 0\hfill & 0.0196\hfill & 0.0092\hfill & 0\hfill & 0.0012\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0.0776\hfill & 0.0024\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.144\hfill & 0.006\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.4\hfill \end{array}\right).$$

Clearly, $P^{\prime }$ is upper triangular; $P^{\prime }\in \mathcal{C}({\mathbf{p}}^{\prime },{\mathbf{q}}^{\prime })$ with ${\mathbf{p}}^{\prime }=(0.3,0.04,0.03,0.08,0.15,0.4)$ and ${\mathbf{q}}^{\prime }=(0.18,0.02,$ $0.03,0.18,0.15,0.44).$

By Lemma 2, $P^{\prime }$ can be further optimised to an upper triangular matrix

$$P^{″}=\left(\begin{array}{cccccc}0.18\hfill & 0\hfill & 0\hfill & 0.12\hfill & 0\hfill & 0\hfill \\ 0\hfill & 0.02\hfill & 0\hfill & 0\hfill & 0\hfill & 0.02\hfill \\ 0\hfill & 0\hfill & 0.03\hfill & 0\hfill & 0\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0.06\hfill & 0\hfill & 0.02\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.15\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.4\hfill \end{array}\right).$$

Using Lemma 2 on the following submatrix of $P^{″}$

$$\left(\begin{array}{cc}0.02\hfill & 0.02\hfill \\ 0\hfill & 0.02\hfill \end{array}\right)$$

and then exchanging the positions of its 2nd and 4th rows, we obtain

$$P^{‴}=\left(\begin{array}{cccccc}0.18\hfill & 0\hfill & 0\hfill & 0.12\hfill & 0\hfill & 0\hfill \\ 0\hfill & 0.02\hfill & 0\hfill & 0.06\hfill & 0\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0.03\hfill & 0\hfill & 0\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.04\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.15\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0.4\hfill \end{array}\right).$$

Note that $P^{‴}$ is upper triangular and $P^{‴}\in \mathcal{C}({\mathbf{p}}^{″},{\mathbf{q}}^{″})$ with ${\mathbf{p}}^{″}=(0.3,0.08,0.03,0.04,0.15,0.4)$ and ${\mathbf{q}}^{″}={\mathbf{q}}^{\prime }$.

4. Conclusions

This study achieves the extremal problem of Shannon entropy for joint distributions with given marginals by establishing a localized optimization method to derive minimum-entropy physical structures. We rigorously demonstrate that minimum-entropy couplings necessitate an order-preserving structure (transformable into upper triangular matrixes via permutations), while maximum-entropy coupling aligns with independent distributions, thereby confirming entropy’s role in quantifying system disorder and addressing the gap in system coupling optimization. The proposed order-preserving coupling framework, combined with matrix structure analysis, not only explains its implications for communications and bioinformatics but also provides a novel approach to simplify exact solution computations through the coupled minimum-entropy system. These findings systematically highlight the critical connection between ordinal relationships and entropy optimization, offering both methodological tools and fresh perspectives for understanding ordered structures in complex systems.