Symmetry-Breaking Bifurcations of the Information Bottleneck and Related Problems

Abstract

In this paper, we investigate the bifurcations of solutions to a class of degenerate constrained optimization problems. This study was motivated by the Information Bottleneck and Information Distortion problems, which have been used to successfully cluster data in many different applications. In the problems we discuss in this paper, the distortion function is not a linear function of the quantizer. This leads to a challenging annealing optimization problem, which we recast as a fixed-point dynamics problem of a gradient flow of a related dynamical system. The gradient system possesses an $S_N$ symmetry due to its invariance in relabeling representative classes. Its flow hence passes through a series of bifurcations with specific symmetry breaks. Here, we show that the dynamical system related to the Information Bottleneck problem has an additional spurious symmetry that requires more-challenging analysis of the symmetry-breaking bifurcation. For the Information Bottleneck, we determine that when bifurcations occur, they are only of pitchfork type, and we give conditions that determine the stability of the bifurcating branches. We relate the existence of subcritical bifurcations to the existence of first-order phase transitions in the corresponding distortion function as a function of the annealing parameter, and provide criteria with which to detect such transitions.

1. Introduction

This paper analyzes bifurcations of solutions to constrained optimization problems of the form

$$\begin{array}{c}\hfill \underset{q\in \Delta }{max}F(q,\beta )=\underset{q\in \Delta }{max}\left(\sum _{i=1}^{N}f(q^i,\beta )\right)\end{array}$$

(1)

as a function of a scalar parameter $\beta$ and a quantizer or classifier $q=(q^1,\cdots ,q^N)$ with $q^i\in {\Re }^K$. The real-valued function f is sufficiently smooth, and $\Delta$ is the constraint space of valid quantizers, a convex set of discrete probabilities (simplices).

This type of problem arises in Rate Distortion Theory [1,2], Deterministic Annealing [3] and biclustering [4]. The specific motivations for the abstract problem formulation given in (1) are the Information Bottleneck [5] and Information Distortion [6] functions

$$\begin{array}{c}\hfill \underset{q\in \Delta }{max}F(q,\beta )=\underset{q\in \Delta }{max}\left(D\left(q\right)-\beta I(Y;T)\right).\end{array}$$

(2)

These were proposed in [5,7] to analyze the Markov chain $X\to Y\to T$ in which $X\to Y$, characterized by a probability $p(X,Y)$, is the original system of interest, characterized by its mutual information $I(X;Y)$, and T is a simplification (quantized version of) Y. Here we work mainly with discrete versions of Y and T, with cardinalities $\left|Y\right|=K$ and $\left|T\right|=N$. Typically $N<<K$. $I(Y;T)$ is the mutual information between the K objects in Y and the N clusters in T. The goal is to cluster K objects in Y into N clusters in T given inputs X such that the function F is maximized in ${\left[q^i\right]}_j$; the probability that the jth element of Y is classified as being a member of the cluster with label $i\in T$. We call such a set of conditional probabilities a stochastic quantizer, or just a quantizer, to relate to the vector quantization literature [8]. The annealing parameter $\beta \in [0,\infty )$.

It has been shown that finding hard-clustering solutions to (2) is NP-complete (combinatorial search) when $D\left(q\right)$ is the mutual information $I(X;T)$ [9], as in the Information Bottleneck [5,10,11] and the Information Distortion [7,12,13] methods. Information Bottleneck (IB) approaches are gaining in penetration into multiple scientific and engineering domains [14,15,16,17,18]. As they typically involve the nonlinear optimization problem (2), there is need for optimization methods for such problems that can avoid the rise in complexity implied by the NP-complete hard-clustering solutions [9]. Originally, Tishby et al. [5] approached this problem with an algorithm inspired by the Blahut–Arimoto approach to solving Rate-Distortion types of problems ([2], Chapter 10). The “self-consistent” equations in [5] optimize both the quantizer and the “relevance” distribution $p\left(x\right|t)$. However, unlike the classic Blahut–Arimoto algorithm, which can guarantee convergence to a unique solution to its iterative scheme because of the convex geometry of the two state spaces, the “self-consistent” equations have no such guarantee due to the more-complicated geometry of three convex sets over which the optimization is performed, as also noted in [5]. Accordingly, in this work, we use the original optimization problem (2) over a single variable: the quantizer (conditional probability) $q\left(t\right|y)$. It may be possible that a related Blahut–Arimoto style optimization coupled to the bifurcation structure of its gradient flow discussed here can lead to additional insights into this problem, but we consider this beyond the scope of this particular manuscript.

We have investigated the structure of soft-clustering annealing-type methods that reach the hard-clustering solution in the limit of the annealing parameter [19,20] through a series of bifurcations. A bifurcation in this context is a point that is a solution $(q^{*},{\beta }^{*})$ to (2) such that the number of solutions to (2) changes in a small neighborhood of $(q^{*},{\beta }^{*})$. Because a bifurcation corresponds to a point at which some of the objects Y have just been classified, in the IB literature, a bifurcation is usually referred to as a phase transition. One of the goals of this and related work is to understand why annealing-type algorithms, such as the original optimization heuristics in [5,10], work as well as they do. This can help with designing further optimization heuristics and can assess how close those can get to the global solutions to IB problems. We believe that this amalgamation of optimization theory and dynamical systems theory, as stated in [19,20], can provide a solid foundation with which to address such optimization challenges.

Because of the form (1) of F, it possesses certain symmetries. That is, the value of $F(q,\beta )$ does not change (is invariant) under arbitrary permutations of the vectors $q^i$. In other words, F is $S_N$-invariant. The form (1) further implies that the Hessian $d_q^{2}F\left(q\right)$ is block diagonal with blocks ${\{d_{q^i}^{2}f\left(q^i\right)\}}_{i=1}^N$. These conditions are met by the Information Distortion function [6],

$$\begin{array}{c}\hfill F_H(q,\beta )=H\left(T\right|Y)+\beta I(X,T),\end{array}$$

(3)

where $H\left(T\right|Y)$ is the entropy, and by the cost function used in the original IB method [5],

$$\begin{array}{c}\hfill F_{IB}(q,\beta )=-I(Y,T)+\beta I(X,T),\end{array}$$

(4)

which is the focus of this manuscript. Both the Information Distortion and Informaton Bottleneck problems have the form given in (1) and (2). Importantly, $d^2{F}_{IB}\left(q\right)$ has a “perpetual kernel“ since each block $d^{2}f\left(q^i\right)$ has the eigenpair (0, $q^i$) for every q [20]. In other words, the Hessian $d^{2}F$ is singular for every q and every value of $\beta$. This makes bifurcation detection challenging because bifurcations can usually be detected by identifying isolated singularities of $d^{2}F$. This degeneracy is a consequence of the translational symmetry of $F_{IB}$: if $\mathit{k}\in ker{d}_q^2{F}_{IB}\left(q^{*}\right)$, then $F_{IB}\left(q^{*}\right)=F_{IB}(q^{*}+t\mathit{k})$ for all $t\in \Re$ such that $q^{*}+t\mathit{k}\in \Delta$. At bifurcations of solutions to (4), the translational symmetry never breaks.

To better understand bifurcations of solutions to problems of the form (1), which includes the problems (3) and (4), we consider the gradient flow

$$\left(\begin{array}{c}\dot{q}\\ \dot{\lambda }\end{array}\right)=\nabla \mathcal{L}(q,\lambda ,\beta )$$

Equilibria of this flow correspond to critical points of (1), where $\mathcal{L}$ is the Lagrangian with respect to the constraints imposed by $\Delta$, and $\lambda$ is the vector of Lagrange multipliers.

Previous work showed that when $d^{2}F$ is generically non-singular, as occurs for the Information Distortion (3), then there are isolated singularities of $d^2\mathcal{L}$ that indicate possible bifurcations of solutions to (1). In this case, an $M>1$-dimensional $ker{d}^{2}F$ necessitates an $M-1$-dimensional $ker{d}^2\mathcal{L}$, which admits a bifurcation of solutions to (1) where symmetry breaks from $S_M$ to $S_m\times S_n$ for every $m,n>0$ such that $m+n=M$ [21].

Here we allow $d^{2}F$ and $d^2\mathcal{L}$ to be singular for every $q\in \Delta$, as occurs for the Information Bottleneck (4). That is, the perpetual kernel for $d^{2}F$ implies that $d^2\mathcal{L}$ also has a perpetual kernel $ker{d}^2\mathcal{L}={\mathcal{K}}_p\left(q\right)$, which means that the eigenvalue crossing condition that must occur at a bifurcation (i.e., $d^2\mathcal{L}$ must have a zero eigenvalue at a bifurcation) [20] is never satisfied in ${\mathcal{K}}_p$. There are a few challenges due to the existence of the perpetual kernel (i.e., degeneracy) of the Information Bottleneck that we address in this paper. First, detecting bifurcations may be problematic because one cannot simply monitor the determinant of either $d^{2}F$ or $d^2\mathcal{L}$. Second, the standard theory that assures the existence of bifurcating branches, the Equivariance Branching Lemma, cannot be applied directly. Lastly, the spaces that contain the bifurcating solutions are always at least two-dimensional, which makes tracking the bifurcating solutions problematic.

Here we address two of these three challenges. We show that at a bifurcation, new eigenvalue(s) of $d^2{F}_{IB}$ and $d^2\mathcal{L}$ must cross zero, causing $ker{d}^2\mathcal{L}$ to expand so that $ker{d}^2\mathcal{L}\left(q^{*}\right)={\mathcal{K}}_p\cup {\mathcal{K}}^{*}$, where ${\mathcal{K}}^{*}$ is the span of the eigenvectors with crossing eigenvalues. Instead of detecting bifurcations by the expensive process of monitoring the expansion of $ker{d}^2\mathcal{L}$ (from ${\mathcal{K}}_p$ to ${\mathcal{K}}_p\cup {\mathcal{K}}^{*}$), we give a simple way to check the eigenvalue crossing condition for annealing problems $F=G\left(q\right)+\beta D\left(q\right)$ as in (2) [20]. We prove the existence of the bifurcating branches by adapting the standard proof for the Equivariant Branching Lemma. This newly developed theory guarantees that bifurcating branches exist in ${\mathcal{K}}^{*}$, are generically pitchforks, and that symmetry breaks from $S_M$ to $S_m\times S_n$. Additionally, we give conditions to check whether the pitchforks are subcritical or supercritical, and how stability of the bifurcating branches relates to optimality in the optimization problem (1).

2. Bifurcation Analysis

2.1. Equivariant Branching Lemma

The Equivariant Branching Lemma relates the subgroup structure of a symmetry group $\mathsf{\Gamma }$ with the existence of symmetry-breaking bifurcating branches of equilibria of $\dot{x}=f(x,\beta )$. Observe that we present a version that does not require absolute irreducibility. For a proof see [22] p. 83.

Theorem 1.

(Equivariant Branching Lemma). Let f be a smooth function $f:V\times \Re \to V$ that is Γ-equivariant for a compact Lie group Γ and a Banach space V. Let Σ be an isotropy subgroup of $\mathsf{\Gamma }$ with $dim\mathrm{Fix}\left(\mathsf{\Sigma }\right)=1$. Suppose that $\mathrm{Fix}\left(\mathsf{\Gamma }\right)=\left\{0\right\}$ and the crossing condition $d_{\beta x}^{2}f(0,0){\mathit{x}}_0\ne 0$ for $x_0\in \mathrm{Fix}\left(\mathsf{\Sigma }\right)$. Then there exists a unique smooth solution branch $(t{x}_0,\beta \left(t\right))$ to $f=0$ with isotropy subgroup Σ.

For an arbitrary $\mathsf{\Gamma }$-equivariant system where bifurcation occurs at $(x^{*},{\beta }^{*})$, the requirement in Theorem 1 that the bifurcation occurs at the origin is accomplished by a translation. Assuring that the Jacobian vanishes, $d_{x}f(0,0)=0$, can be effected by restricting and projecting the system onto the kernel of the Jacobian. This transform is called the Liapunov–Schmidt reduction (see [23]).

The Equivariant Branching Lemma does not directly apply to yield bifurcating branches for the problem (1) at q for which $d^{2}F$ is singular for the following reasons:

• ${\mathcal{K}}_p$ and ${\mathcal{K}}^{*}$ have independent bases, which implies that each is invariant to the action of $S_N$, and so the decomposition $ker{d}^2\mathcal{L}\left(q^{*}\right)={\mathcal{K}}_p\times {\mathcal{K}}^{*}$ shows that $S_N$ does not act absolutely irreducibly on $ker{d}^{2}F\left(q^{*}\right)$, but it does act absolutely irreducibly on each of these disjoint subspaces separately. This is why we present a version of the Equivariant Branching Lemma that does not require absolute irreducibility.
• The Liapunov–Schmidt reduction onto $ker{d}^2\mathcal{L}\left(q^{*}\right)$ is clear, but not onto ${\mathcal{K}}^{*}$.
• $\mathrm{Fix}(S_m\times S_n)\cap ker{d}^2\mathcal{L}\left(q^{*}\right)$ is two-dimensional with basis

  $$\left\{\right(n\mathit{v},\cdots ,n\mathit{v},-m\mathit{v},\cdots ,-m\mathit{v}),(n\mathit{y},\cdots ,n\mathit{y},-m\mathit{y},\cdots ,-m\mathit{y}\left)\right\},$$

  where $\mathit{v},\mathit{y}\in {\Re }^K$.

We address these issues in the manuscript and show that a small modification of the Equivariant Branching Lemma allows for similar analysis to be successfully applied to Information Bottleneck-style problems such as (2) with minimal modifications to the original algorithm from [20].

2.2. A Gradient Flow

We now lay the groundwork necessary to determine the bifurcations of local solutions to (1)

$$\underset{q\in \Delta }{max}F(q,\beta ),$$

where $F={\sum }_{i=1}^{N}f(q^i,\beta )$, which includes as a special case the Information Distortion (3) and Information Bottleneck (4) problems. The convex set of discrete conditional probabilities is

$$\Delta :=\left\{q\in {\Re }^{NK}|\sum _{i=1}^N{q}_k^i=1\forall k:1\le k\le K \mathrm{and} q_k^i\ge 0\forall i,k\right\}.$$

Due to the form of F, it has the following properties:

• $F(q,\beta )$ is an $S_N$-invariant, real-valued function of q, where the action of $S_N$ on q permutes the component vectors $q^i$, $i=1,\dots ,N$, of $q\in \Delta$.
• The $NK\times NK$ Hessian $d_q^{2}F(q,\beta )$ is block diagonal, where the ith $K\times K$ block is $d^{2}f\left(q^i\right)$.

The Lagrangian of (1) with respect to the equality constraints from $\Delta$ is

$$\begin{array}{c}\hfill \mathcal{L}(q,\lambda ,\beta )=F(q,\beta )+\sum _{k=1}^K{\lambda }_k\left(\sum _{i=1}^N{q}_k^i-1\right).\end{array}$$

(5)

The scalar ${\lambda }_k$ is the Lagrange multiplier for the constraint ${\sum }_{i=1}^N{q}_k^i-1=0$, and $\lambda \in {\Re }^K$ is the vector of Lagrange multipliers $\lambda ={({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_K)}^T.$ The gradient of the Lagrangian in (5) is

$$\nabla \mathcal{L}:={\nabla }_{q,\lambda }\mathcal{L}(q,\lambda ,\beta )=\left(\begin{array}{c}{\nabla }_q\mathcal{L}\\ {\nabla }_{\lambda }\mathcal{L}\end{array}\right),$$

where ${\nabla }_q\mathcal{L}=\nabla F(q,\beta )+\mathsf{\Lambda }$ and $\mathsf{\Lambda }={\left({\lambda }^T,{\lambda }^T,\cdots {\lambda }^T\right)}^T\in {\mathbf{R}}^{NK}$. The gradient ${\nabla }_{\lambda }\mathcal{L}$ is a vector of K constraints

$$\begin{array}{c}\hfill {\nabla }_{\lambda }\mathcal{L}=\left(\begin{array}{c}\sum _i{q}_1^i-1\\ \sum _i{q}_2^i-1\\ ⋮\\ \sum _i{q}_K^i-1\end{array}\right).\end{array}$$

Let J be the Jacobian of $d_q{\nabla }_{\lambda }\mathcal{L}$

$$\begin{array}{c}\hfill J:=d_q{\nabla }_{\lambda }\mathcal{L}=\underset{N \mathrm{blocks}}{\underbrace{\left(\begin{array}{cccc}{I}_K& I_K& \cdots & I_K\end{array}\right)}}.\end{array}$$

(6)

Observe that J has full row rank. The Hessian of (5) with respect to the vector $\left(\begin{array}{c}q\\ \lambda \end{array}\right)\in {\Re }^{NK+K}$ is

$$\begin{array}{c}\hfill d^2\mathcal{L}\left(q\right):=d^2\mathcal{L}(q,\lambda ,\beta )=\left(\begin{array}{cc}{d}^{2}F(q,\beta )& J^T\\ J& \mathbf{0}\end{array}\right),\end{array}$$

(7)

where $\mathbf{0}$ is $K\times K$. The $NK\times NK$ matrix $d^{2}F\left(q\right):=d_q^{2}F(q,\beta )$ is the block diagonal Hessian of F with $K\times K$ blocks ${\left\{d^{2}f(q^i,\beta )\right\}}_{i=1}^N$.

The dynamical system whose equilibria are stationary points of (1) is the gradient flow of the Lagrangian

$$\begin{array}{c}\hfill \left(\begin{array}{c}\dot{q}\\ \dot{\lambda }\end{array}\right)=\nabla \mathcal{L}(q,\lambda ,\beta )\end{array}$$

(8)

for $\mathcal{L}$ as defined in (5) and $\beta \in [0,\infty )$. The equilibria of (8) are points $\left(\begin{array}{c}{q}^{*}\\ {\lambda }^{*}\end{array}\right)\in {\mathbf{R}}^{NK+K}$ where

$$\nabla \mathcal{L}(q^{*},{\lambda }^{*},\beta )=0.$$

The Jacobian of this system is the Hessian $d^2\mathcal{L}(q,\lambda ,\beta )$ from (7).

Remark 1.

By the theory of constrained optimization [24], the equilibria $(q^{*},{\lambda }^{*},\beta )$ of (8) where $d^{2}F(q^{*},\beta )$ is negative definite on $kerJ$ are local solutions of (1). Conversely, if $(q^{*},\beta )$ is a local solution of (1), then there exists a vector of Lagrange multipliers ${\lambda }^{*}$ so that $(q^{*},{\lambda }^{*},\beta )$ is an equilibrium of (8) (this necessary requirement is called the Karush–Kuhn–Tucker conditions) such that $d^{2}F(q^{*},\beta )$ is non-positive definite on $kerJ$.

2.3. Equilibria with Symmetry

Next, we categorize the equilibria of (8) according to their symmetries, which allows us to determine when to expect symmetry-breaking bifurcations.

Let $q\in \mathrm{Fix}\left(S_M\right)$ for some $1\le M\le N$. Then there exists a partition of $\{1,2,\dots ,N\}$ into the sets $\mathcal{U}$ and $\mathcal{R}$, where $\left|\mathcal{U}\right|=M$, so that $q^i=q^j$ if and only if $i,j\in \mathcal{U}$. Clearly, $d^{2}F$ has M identical blocks, ${\left\{d^f\left(q^i\right)\right\}}_{i\in \mathcal{U}}$.

To ease the notation, and without loss of generality, we set

$$\mathcal{U}:=\{1,\dots ,M\}\mathrm{and}\mathcal{R}:=\{M+1,\dots ,N\}.$$

To distinguish between the blocks of $d^{2}F$, we write

$$\begin{array}{c}\hfill B:=d^{2}f\left(q^i\right) \mathrm{for} \le 1\le i\le M \mathrm{and} R_i:=d^{2}f\left(q^i\right) \mathrm{for} M+1\le i\le N.\end{array}$$

(9)

As mentioned in the introduction, we assume that for each $q\in \Delta$, each block $d^{2}f\left(q^i\right)$ always has at least a one-dimensional kernel with basis vector(s) which depend on q. Thus, $dimker{d}^{2}F\ge N$. At an equilibrium of $(q^{*},{\lambda }^{*},{\beta }^{*})$ of (8) where $q\in \mathrm{Fix}\left(S_M\right)$, we consider the following three cases:

• $dimker{d}^{2}F\left(q^{*}\right)>N+1$;
• $dimker{d}^{2}F\left(q^{*}\right)=N+1$;
• $dimker{d}^{2}F\left(q^{*}\right)=N$.

We will show that the first case necessitates a symmetry-breaking bifurcation (Theorem 3). In the second case, there is no bifurcation (Corollary 1). Finally, in the third case, we expect a saddle node [21], a symmetry-preserving bifurcation.

We are able to distinguish between the three cases above by considering which blocks of $d^{2}F\left(q^{*}\right)$ have kernels that have more than one dimension. This motivates the following definition.

Definition 1.

An equilibrium $(q^{*},{\lambda }^{*},{\beta }^{*})$ of (8) is M-singular (or, equivalently, $q^{*}$ is M-singular) if:

• $q\in \mathrm{Fix}\left(S_M\right)$ so that $q^i=q^j$ for every $1\le i,j\le M$.
• For B, the M block(s) of the Hessian defined in (9), $kerB$ has dimension 2 with basis vectors $\mathit{v},\mathit{y}\in {\Re }^K$. $\mathit{v}$ is associated with the crossing eigenvalues, and $\mathit{y}$ is associated with the constant zero eigenvalue of B.
• The $N-M$ block(s) of the Hessian ${\left\{R_i\right\}}_{i\in \Re }$, defined in (9), each have a one-dimensional kernel with basis vector $\mathit{z}\left(i\right)\in {\Re }^K$.
• The vectors $\mathit{v}$, $\mathit{y}$ and $\left\{\mathit{z}\right(i\left)\right\}$ are linearly independent.
• The matrix

  $$\begin{array}{c}\hfill A:=B\sum _{i=M+1}^N{R}_i^{-}+M{I}_K\end{array}$$

  (10)

  is nonsingular. $R_i^{-}$ is the Moore–Penrose inverse of $R_i$. When $M=N$, we define $A:=N{I}_K$.

We wish to emphasize that we showed in [21] that requirements 2–5 in Definition 1 hold generically.

A straightforward calculation shows that every block of the Hessian $d^{2}F$ of the Information Bottleneck cost function (2) is singular for every $(q,\beta )$, and the basis for $ker{d}^{2}f\left(q^i\right)$ is $\mathit{y}=q^i$ for $1\le i\le M$ and $\mathit{z}\left(i\right)=q^i$ for $M+1\le i\le N$ (Lemma 42 in [25]), which assures that these vectors are linearly independent, as in Definition 1.4. At a bifurcation, the kernels of the identical blocks B expand by $\mathit{v}$ as in Definition 1.2. Using the notation above, $\mathit{y}=q^i$ for each $i\in \mathcal{U}$, and $\mathit{z}\left(i\right)=q^i$ for each $i\in \mathcal{R}$.

2.4. The Kernel at a Bifurcation

The equilibria of (8) change their stability with $\beta$, and hence change the solutions to (1). The changes of stability are determined by the kernel of $d^2\mathcal{L}\left(q^{*}\right)$ at a bifurcation point $q^{*}$. In this section we show that for any $q\in \mathrm{Fix}\left(S_M\right)$ with $M>1$, $d^2\mathcal{L}\left(q^{*}\right)$ has a perpetual kernel ${\mathcal{K}}_p$ that is at least $M-1$ dimensional. The zero eigenvalues associated with the eigenvectors in ${\mathcal{K}}_p$ remain constant, so that at a bifurcation point $(q^{*},{\lambda }^{*},{\beta }^{*})$ of (8) where $q^{*}$ is M-singular, new eigenvalues of $d^2\mathcal{L}$ must cross zero. Thus, the kernel expands, and the bifurcating directions exist in an “expanded” kernel of $d^2\mathcal{L}\left(q^{*}\right)$, $ker{d}^2\mathcal{L}\left(q^{*}\right)={\mathcal{K}}^{*}\times {\mathcal{K}}_p$.

We determine a basis for $ker{d}^2\mathcal{L}$ at an M-singular $q^{*}$ when $M>1$. If q is 1-singular with a trivial isotropy group (i.e., no symmetery), then $d^2\mathcal{L}\left(q^{*}\right)$ is non-singular—${\mathcal{K}}_p$ disappears. First, we ascertain a basis for $ker{d}^{2}F\left(q^{*}\right)$.

Recall that in the preliminaries, when $\mathit{x}\in {\Re }^{NK}$, we defined ${\mathit{x}}^j\in {\mathbf{R}}^K$ to be the jth vector component of $\mathit{x}$. We now define the linearly independent vectors ${\left\{{\mathit{v}}_i\right\}}_{i=1}^M$, ${\left\{{\mathit{y}}_i\right\}}_{i=1}^M$, and ${\left\{{\mathit{z}}_k\right\}}_{k=M+1}^N$ in ${\Re }^{NK}$ by

$$\begin{array}{ccc}\hfill {\mathit{v}}_i^j:=\left\{\begin{array}{c}\mathit{v} \mathrm{if} 1\le i=j\le M\hfill \\ \mathbf{0} \mathrm{otherwise}\hfill \end{array}\right.,& {\mathit{y}}_i^j:=\left\{\begin{array}{c}\mathit{y} \mathrm{if} 1\le i=j\le M\hfill \\ \mathbf{0} \mathrm{otherwise}\hfill \end{array}\right.,\hfill & \hfill {\mathit{z}}_k^j:=\left\{\begin{array}{c}\mathit{z}\left(i\right) \mathrm{if} M+1\le j=k\le N\hfill \\ \mathbf{0} \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(11)

where $\mathbf{0}\in {\Re }^K$, and $\mathit{v}$ and $\mathit{y}$ are defined in Definition 1.2. For example, if $M=2$ and $N=3$, then ${\mathit{v}}_1:={({\mathit{v}}^T,\mathbf{0},\mathbf{0})}^T$ and ${\mathit{v}}_2:={(\mathbf{0},{\mathit{v}}^T,\mathbf{0})}^T$.

Due to the block diagonal form of $d^{2}F\left(q^{*}\right)$, it is easy to see that the $N+M$ vectors defined in (11) form a basis for $ker{d}^{2}F\left(q^{*}\right)$.

Now, let

$$\begin{array}{ccc}\hfill V_i=\left(\begin{array}{c}{\mathit{v}}_i\\ \mathbf{0}\end{array}\right)-\left(\begin{array}{c}{\mathit{v}}_M\\ \mathbf{0}\end{array}\right),& Y_i=\left(\begin{array}{c}{\mathit{y}}_i\\ \mathbf{0}\end{array}\right)-\left(\begin{array}{c}{\mathit{y}}_M\\ \mathbf{0}\end{array}\right),& Z_k=\left(\begin{array}{c}{\mathit{z}}_k\\ \mathbf{0}\end{array}\right)-\left(\begin{array}{c}{\mathit{z}}_N\\ \mathbf{0}\end{array}\right)\hfill \end{array}$$

(12)

for $i=1,\cdots ,M-1$ and $M+1\le k\le N-1$ where $\mathbf{0}\in {\Re }^K$. From (7), it is easy to see that these three sets of vectors are in $ker{d}^2\mathcal{L}\left(q^{*}\right)$. The next theorem shows that ${\left\{V_i\right\}}_{i=1}^{M-1}\bigcup {\left\{Y_i\right\}}_{i=1}^{M-1}$ are a basis for $ker{d}^2\mathcal{L}\left(q^{*}\right)$. This natural partition of the basis vectors shows that $ker{d}^2\mathcal{L}\left(q^{*}\right)$ can be written as $ker{d}^2\mathcal{L}\left(q^{*}\right)={\mathcal{K}}_p\times {\mathcal{K}}^{*}$. According to Definition 1, the “perpetual kernel” corresponding to constant zero eigenvalues of $d^2\mathcal{L}\left(q^{*}\right)$ is generated by

$${\mathcal{K}}_p=<{\left\{Y_i\right\}}_{i=1}^{M-1}>.$$

The part of the kernel that arises at a bifurcation corresponding to eigenvalues crossing zero is

$${\mathcal{K}}^{*}=<{\left\{V_i\right\}}_{i=1}^{M-1}>.$$

The vectors $\left\{Z_k\right\}$ do not contribute to $ker{d}^2\mathcal{L}\left(q^{*}\right)$.

Theorem 2.

If $q^{*}$ is M-singular for $1<M\le N$, then $\left\{V_i\right\}\bigcup \left\{Y_i\right\}$ from (12) are a basis for $ker{d}^2\mathcal{L}\left(q^{*}\right)$.

Proof. 

To show that ${\left\{V_i\right\}}_{i=1}^{M-1}\bigcup {\left\{Y_i\right\}}_{i=1}^{M-1}$ span $ker{d}^2\mathcal{L}\left(q^{*}\right)$, let $\mathit{k}\in ker{d}^2\mathcal{L}\left(q^{*}\right)$ and decompose it as

$$\begin{array}{c}\hfill \mathit{k}=\left(\begin{array}{c}{\mathit{k}}_F\\ {\mathit{k}}_J\end{array}\right)\end{array}$$

(13)

where ${\mathit{k}}_F$ is $NK\times 1$, and ${\mathit{k}}_J$ is $K\times 1$. Hence,

$$\begin{array}{ccc}& d^2\mathcal{L}(q^{*},{\lambda }^{*},\beta )\mathit{k}=\left(\begin{array}{cc}{d}^{2}F(q^{*},{\beta }^{*})& J^T\\ J& \mathbf{0}\end{array}\right)\left(\begin{array}{c}{\mathit{k}}_F\\ {\mathit{k}}_J\end{array}\right)=\mathbf{0}& \hfill \\ & \Rightarrow d^{2}F(q^{*},\beta ){\mathit{k}}_F=-J^T{\mathit{k}}_J& \hfill \\ \hfill & J{\mathit{k}}_F=\mathbf{0}.\end{array}$$

(14)

Now, from (6) and the fact that $d^{2}F$ is block diagonal, we have

$$\begin{array}{c}\hfill \left(\begin{array}{cccc}{d}^{2}f\left(q^1\right)& \mathbf{0}& \cdots & \mathbf{0}\\ \mathbf{0}& d^{2}f\left(q^2\right)& \cdots & \mathbf{0}\\ ⋮& ⋮& & ⋮\\ \mathbf{0}& \mathbf{0}& \cdots & d^{2}f\left(q^N\right)\end{array}\right){\mathit{k}}_F=-\left(\begin{array}{c}{\mathit{k}}_J\\ {\mathit{k}}_J\\ ⋮\\ {\mathit{k}}_J\end{array}\right).\end{array}$$

(15)

We set

$$\begin{array}{c}\hfill {\mathit{k}}_F:={({\mathit{x}}_1^T{\mathit{x}}_2^T\dots {\mathit{x}}_N^T)}^T,\end{array}$$

(16)

and using the notation from (9), then (15) implies

$$\begin{array}{ccc}\hfill B{\mathit{x}}_i& =& -{\mathit{k}}_J \mathrm{for} 1\le i\le M\hfill \\ \hfill R_i{\mathit{x}}_i& =& -{\mathit{k}}_J \mathrm{for} M+1\le i\le N.\hfill \end{array}$$

(17)

It follows that ${\mathit{x}}_i=R_i^{-}B{\mathit{x}}_1$ for every $M+1\le i\le N$. By (14), we have that ${\sum }_{i=1}^N{\mathit{x}}_i=\mathbf{0}$, and so

$$\begin{array}{ccc}& {\sum }_{i=1}^M{\mathit{x}}_i+{\sum }_{i=M+1}^N{\mathit{x}}_i=\mathbf{0}& \\ & \Rightarrow {\sum }_{i=1}^M{\mathit{x}}_i+{\sum }_{i=M+1}^N{R}_i^{-}B{\mathit{x}}_1+=\mathbf{0}.& \hfill \end{array}$$

By (17), for every $1\le i\le M,{\mathit{x}}_i$ can be written as ${\mathit{x}}_i={\mathit{x}}_p+d_i\mathit{v}+e_i\mathit{y}$, where ${\mathit{x}}_p\in \mathrm{range}\left(B\right)$, $d_{\eta },e_{\eta }\in \Re$, and $\mathit{v}$ and $\mathit{y}$ are the basis vectors of $kerB$ from Definition 1.2. Thus,

$$\begin{array}{ccc}\hfill B\sum _{i=1}^M({\mathit{x}}_p+d_i\mathit{v}+e_i\mathit{y})& +& B\sum _{i=M+1}^N{R}_i^{-}B({\mathit{x}}_p+d_1\mathit{v}+e_1\mathit{y})=\mathbf{0}\hfill \\ & \iff & (B\sum _{i=M+1}^N{R}_i^{-}+M{I}_K)B{\mathit{x}}_p=\mathbf{0}\hfill \\ & \iff & B{\mathit{x}}_p=\mathbf{0}\hfill \end{array}$$

since $A=B{\sum }_{i=M+1}^N{R}_i^{-}+M{I}_K$ is nonsingular. This shows that ${\mathit{x}}_p=\mathbf{0}$. Therefore, ${\mathit{x}}_i=d_i\mathit{v}+e_i\mathit{y}$ for every $1\le i\le M$. Now (17) shows that ${\mathit{k}}_J=\mathbf{0}$, and so ${\mathit{x}}_i\in ker{R}_i$ for $M+1\le i\le N$, which implies that

$$\begin{array}{c}\hfill {\mathit{x}}_i=c_i\mathit{z}\left(i\right) \mathrm{for} M+1\le i\le N.\end{array}$$

Hence, $\mathit{k}=\left(\begin{array}{c}{\mathit{k}}_F\\ \mathbf{0}\end{array}\right)$, where ${\mathit{k}}_F^i=\left\{\begin{array}{c}{d}_i\mathit{v}+e_i\mathit{y} \mathrm{if} 1\le i\le M\\ c_i\mathit{z}\left(i\right) \mathrm{if} M+1\le i\le N\end{array}\right.$, from which it follows that

$$\begin{array}{c}\hfill J{\mathit{k}}_F=\sum _{i=1}^N{\mathit{x}}_i=\sum _{i=1}^M{d}_i\mathit{v}+\sum _{i=1}^M{e}_i\mathit{y}+\sum _{i=M+1}^N{c}_i\mathit{z}\left(i\right)=\mathbf{0}.\end{array}$$

(18)

Linear independence (Definition 1.4) implies that $\sum d_i=\sum e_i=d_i=0$. Thus, ${\mathit{k}}_F={\sum }_{i=1}^{M-1}{d}_i({\mathit{v}}_i-{\mathit{v}}_M)+{\sum }_{i=1}^{M-1}{e}_i({\mathit{y}}_i-{\mathit{y}}_M).$ Therefore, the linearly independent vectors $\left\{V_i\right\}=\{\left(\begin{array}{c}{\mathit{v}}_i-{\mathit{v}}_M\\ \mathbf{0}\end{array}\right)\}$ and $\left\{Y_i\right\}=\{\left(\begin{array}{c}{\mathit{y}}_i-{\mathit{y}}_M\\ \mathbf{0}\end{array}\right)\}$ span $ker{d}^2\mathcal{L}\left(q^{*}\right)$. □

Corollary 1.

If $q^{*}$ is 1-singular and has isotropy group equal to the identity, then $d^2\mathcal{L}\left(q^{*}\right)$ is nonsingular.

Proof. 

If q is 1-singular, then $d^{2}F\left(q^{*}\right)$ has a single block B with a two-dimensional kernel. The other $N-1$ blocks $\left\{R_i\right\}$ are distinct with one-dimensional kernels. By constructing the vectors as in (11), we see that $dimker{d}^{2}F\left(q^{*}\right)=N+1$ with basis vectors ${\mathit{v}}_1,{\mathit{y}}_1,$${\left\{{\mathit{z}}_i\right\}}_{i=2}^N$. Now, following the proof of Theorem 2, we take an arbitrary $\mathit{k}\in ker{d}^2\mathcal{L}(q^{*},\lambda ,\beta ),$ and then decompose $\mathit{k}$ as in (13) and (16). The proof to Theorem 2 holds for the present case up until, and including (18). Linear independence now shows that $d_i=e_i=c_i=0$, which implies that $\mathit{k}=\mathbf{0}$. □

Remark 2. 

The independent bases given for ${\mathcal{K}}_p$ and ${\mathcal{K}}^{*}$ in Theorem 2 imply that each is invariant to the action of $S_N$, and so the decomposition $ker{d}^2\mathcal{L}\left(q^{*}\right)={\mathcal{K}}_p\times {\mathcal{K}}^{*}$ shows that $S_N$ does not act absolutely irreducibly on $ker{d}^{2}F\left(q^{*}\right)$. That is, by definition,

$$d_{\mathit{x}}r(\mathbf{0},\beta )\ne c\left(\beta \right)I_{2M-2}.$$

The explicit bases show that ${\mathcal{K}}_p,\mathcal{K}\cong \{\mathit{x}\in {\mathbf{R}}^M:\sum {\left[\mathit{x}\right]}_i=0\}$, which implies that $S_M$ acts absolutely irreducibly on ${\mathcal{K}}_p$ and ${\mathcal{K}}^{*}$ [26]. Thus, ${\mathcal{K}}_p$ and ${\mathcal{K}}^{*}$ are each $S_M$-irreducible.

2.5. Liapunov–Schmidt Reduction

To show the existence of bifurcating branches from a bifurcation point $(q^{*},{\lambda }^{*},{\beta }^{*})$ of equilibria of (8), the Equivariant Branching Lemma requires that the bifurcation is translated to $(\mathbf{0},\mathbf{0},0)$ and that the Jacobian vanishes at bifurcation. To accomplish the former, consider

$$\begin{array}{c}\hfill \mathcal{F}(q,\lambda ,\beta ):=\nabla \mathcal{L}(q+q^{*},\lambda +{\lambda }^{*},\beta +{\beta }^{*}).\end{array}$$

To assure that the Jacobian vanishes, we restrict and project $\mathcal{F}$ onto $ker{d}^2\mathcal{L}\left(q^{*}\right)$ in a neighborhood of $(\mathbf{0},\mathbf{0},0)$. This is the Liapunov–Schmidt reduction of $\mathcal{F}$ [23],

$$\begin{array}{ccc}\hfill r& :& {\mathbf{R}}^{M-1}\times \mathbf{R}\to {\mathbf{R}}^{M-1}\hfill \\ \hfill r(\mathit{x},\beta )& =& W^T(I-E)\mathcal{F}(W\mathit{x}+U(W\mathit{x},\beta ),\beta )\hfill \end{array}$$

(19)

where $W\mathit{x}+U(W\mathit{x},\beta )=\left(\begin{array}{c}q\\ \lambda \end{array}\right)$. The $(NK+K)\times (NK+K)$ matrix $I-E$ is the projection matrix onto $ker\mathcal{F}(\mathbf{0},0)=ker{d}^2\mathcal{L}\left(q^{*}\right)$ with $ker(I-E)=\mathrm{range}{d}^2\mathcal{L}\left(q^{*}\right)$. W is the $(NK+K)\times (2M-2)$ matrix whose columns are the basis vectors $\left\{V_i\right\}\cup \left\{Y_i\right\}$ of $ker{d}^2\mathcal{L}\left(q^{*}\right)$ from (12) so that $W\mathit{x}$ is a vector in $ker{d}^2\mathcal{L}\left(q^{*}\right)$. The vector function $U(W\mathit{x},\beta )$ is the component of $(q,\lambda )$ that is in range $d^2\mathcal{L}\left(q^{*}\right)$ such that $E\mathcal{F}(W\mathit{x}+U(\mathit{x},\beta ),\beta )=\mathbf{0}$, $U(\mathbf{0},0)=\mathbf{0}$, and

$$\begin{array}{c}\hfill d_{\mathit{x}}U(\mathbf{0},0)=\mathbf{0}.\end{array}$$

(20)

The system defined by the Liapunov–Schmidt reduction, $\dot{\mathit{x}}=r(\mathit{x},\beta )$, has a bifurcation of equilibria at $(\mathit{x}=\mathbf{0},\beta =0)$, which are in $1-1$ correspondence with equilibria of (8). However, the stability of these associated equilibria is not necessarily the same.

It is straightforward to verify the following derivatives ([23] p. 32), which we will require in the sequel. The $(2M-2)\times (2M-2)$ Jacobian of (19) is

$$\begin{array}{c}\hfill d_{\mathit{x}}r(\mathit{x},\beta )=W^T(I-E)d_{q,\lambda }^2\mathcal{L}(q+q^{*},\lambda +{\lambda }^{*},\beta +{\beta }^{*})(W+d_{\mathit{x}}U(W\mathit{x},\beta )),\end{array}$$

(21)

which shows that

$$\begin{array}{c}\hfill d_{\mathit{x}}r(\mathbf{0},0)=\mathbf{0}\end{array}$$

(22)

since $ker(I-E)=\mathrm{range}{d}^2\mathcal{L}\left(q^{*}\right)$.

Our crossing condition at a bifurcation depends on the matrix of derivatives

$$\begin{array}{c}\hfill \frac{{\partial }^2{r}_i}{\partial \beta \partial x_j}(\mathbf{0},0)=d_{\beta }{d}^2\mathcal{L}[{\mathit{w}}_i,{\mathit{w}}_j]-d^3\mathcal{L}[{\mathit{w}}_i,{\mathit{w}}_j,L^{-}{d}_{\beta }\nabla \mathcal{L}]\end{array}$$

(23)

where the derivatives of $\mathcal{L}$ are evaluated at $(q^{*},{\lambda }^{*},{\beta }^{*})$, and $L^{-}$ is the Moore–Penrose-generalized inverse [27] of $d^2\mathcal{L}\left(q^{*}\right)$. The vectors ${\left\{{\mathit{w}}_i\right\}}_{i=1}^{2M-2}$ are the basis vectors of $ker{d}^2\mathcal{L}\left(q^{*}\right)$ from Theorem 2.

The $(2M-2)\times (2M-2)\times (2M-2)$ three-dimensional array of second derivatives is

$$\begin{array}{ccc}\hfill \frac{{\partial }^2{r}_i}{\partial x_j\partial x_k}(\mathbf{0},0)& =& d^3\mathcal{L}(q^{*},{\lambda }^{*},{\beta }^{*})[{\mathit{w}}_i,{\mathit{w}}_j,{\mathit{w}}_k].\hfill \end{array}$$

In [21], we showed that $\frac{{\partial }^2{r}_i}{\partial x_j\partial x_k}(\mathbf{0},0)=0$ whenever $i=j=k\le M-1$. In the present case, there are more zero entries since now the basis vectors $\left\{{\mathit{w}}_i\right\}$ are of two types: ${\mathit{w}}_i=V_i$ for $1\le i\le M-1$ (basis vectors of ${\mathcal{K}}^{*}$); or ${\mathit{w}}_i=Y_{i-M+1}$ for $M\le i\le 2M-2$ (basis vectors of ${\mathcal{K}}_p$, see (12)). We now consider the case when $i,j\le M-1$ and $k>M-1$. All other cases are dealt with using a similar argument. Substituting in for ${\mathit{w}}_i$ we have

$$\begin{array}{ccc}\hfill \frac{{\partial }^2{r}_i}{\partial x_j\partial x_k}(\mathbf{0},0)& =& \sum _{\nu ,\delta ,\eta =1}^N\sum _{l,m,n=1}^K\frac{{\partial }^{3}F(q^{*},{\beta }^{*})}{\partial q_l^{\nu }\partial q_m^{\delta }\partial q_n^{\eta }}{[{\mathit{v}}_i-{\mathit{v}}_M]}_l^{\nu }{[{\mathit{v}}_j-{\mathit{v}}_M]}_m^{\delta }{[{\mathit{y}}_{k-M+1}-{\mathit{y}}_M]}_n^{\eta }\hfill \\ & =& \sum _{l,m,n=1}^K\frac{{\partial }^{3}f(q^{\nu }*,{\beta }^{*})}{\partial q_l^{\nu }\partial q_m^{\nu }\partial q_n^{\nu }}\left({\delta }_{ij(k-M+1)}{\left[\mathit{v}\right]}_l{\left[\mathit{v}\right]}_m{\left[\mathit{y}\right]}_n-{\left[\mathit{v}\right]}_l{\left[\mathit{v}\right]}_m{\left[\mathit{y}\right]}_n\right).\hfill \end{array}$$

(24)

The vectors $\mathit{v}$ and $\mathit{y}$ are defined in (2). An immediate consequence of this calculation is that $\frac{{\partial }^2{r}_i}{\partial x_j\partial x_k}(\mathbf{0},0)=0$ whenever $i=j=k-M+1$. Thus, similar arguments show that $\frac{{\partial }^2{r}_i}{\partial x_j\partial x_k}(\mathbf{0},0)=0$ whenever:

• $i=j=k$;
• $i-M+1=j=k$, $i=j-M+1=k$, $i=j=k-M+1$;
• $i-M+1=j-M+1=k$, $i-M+1=j=k-M+1$, $i=j-M+1=k-M+1$.

Further, we get four different “cubes” of identical entries in the 3-D array. They are:

• For $i,j,k\le M-1$, not all equal, the value of the cube is

  $$-\sum _{l,m,n=1}^K\frac{{\partial }^{3}f(q^{\nu }*,{\beta }^{*})}{\partial q_l^{\nu }\partial q_m^{\nu }\partial q_n^{\nu }}{\left[\mathit{v}\right]}_l{\left[\mathit{v}\right]}_m{\left[\mathit{v}\right]}_n;$$
• For $i,j\le M-1$, not both equal, and $j>M-1$, the value of the cube is

  $$-\sum _{l,m,n=1}^K\frac{{\partial }^{3}f(q^{\nu }*,{\beta }^{*})}{\partial q_l^{\nu }\partial q_m^{\nu }\partial q_n^{\nu }}{\left[\mathit{v}\right]}_l{\left[\mathit{v}\right]}_m{\left[\mathit{y}\right]}_n;$$
• For $i\le M-1$ and $j,k>M-1$, not both equal, the value of the cube is

  $$-\sum _{l,m,n=1}^K\frac{{\partial }^{3}f(q^{\nu }*,{\beta }^{*})}{\partial q_l^{\nu }\partial q_m^{\nu }\partial q_n^{\nu }}{\left[\mathit{v}\right]}_l{\left[\mathit{y}\right]}_m{\left[\mathit{y}\right]}_n;$$
• For $i,j,k>M-1$, not all equal, the value of the cube is

  $$-\sum _{l,m,n=1}^K\frac{{\partial }^{3}f(q^{\nu }*,{\beta }^{*})}{\partial q_l^{\nu }\partial q_m^{\nu }\partial q_n^{\nu }}{\left[\mathit{y}\right]}_l{\left[\mathit{y}\right]}_m{\left[\mathit{y}\right]}_n.$$

The points above will prove useful when proving that $d^{2}r(\mathbf{0},0)=\mathbf{0}$.

The four-dimensional array of third derivatives of r is

$$\begin{array}{ccc}\hfill \frac{{\partial }^3{r}_i}{\partial x_j\partial x_k\partial x_l}(\mathbf{0},0)=d^4\mathcal{L}[{\mathit{w}}_i,{\mathit{w}}_j,{\mathit{w}}_k,{\mathit{w}}_l]& -& d^3\mathcal{L}[{\mathit{w}}_i,{\mathit{w}}_j,L^{-}{d}^3\mathcal{L}[{\mathit{w}}_k,{\mathit{w}}_l]]\hfill \\ & -& d^3\mathcal{L}[{\mathit{w}}_i,{\mathit{w}}_k,L^{-}{d}^3\mathcal{L}[{\mathit{w}}_j,{\mathit{w}}_l]]\hfill \\ & -& d^3\mathcal{L}[{\mathit{w}}_i,{\mathit{w}}_l,L^{-}{d}^3\mathcal{L}[{\mathit{w}}_j,{\mathit{w}}_k]]\hfill \end{array}$$

(25)

where the derivatives of $\mathcal{L}$ are evaluated at $(q^{*},{\lambda }^{*},{\beta }^{*})$, and $L^{-}$ is the Moore–Penrose-generalized inverse [27] of $d^2\mathcal{L}\left(q^{*}\right)$.

Since $ker{d}^2\mathcal{L}\left(q^{*}\right)$ is not absolutely irreducible, but ${\mathcal{K}}^{*}$ is, one might try to define a Liapunov–Schmidt reduction by restricting and projecting $\nabla \mathcal{L}$ onto ${\mathcal{K}}^{*}$. One issue with projecting the reduction onto ${\mathcal{K}}^{*}$ is how to define the projection matrix E so that

$$E\mathcal{F}=0 \mathrm{and} (I-E)\mathcal{F}=0 \mathrm{if} \mathrm{and} \mathrm{only} \mathrm{if} \mathcal{F}=0$$

holds and $E{d}_{\mathit{x}}r(\mathbf{0},0)$ is non-singular in $\mathrm{range}\left(E\right)$ so that the Implicit Function Theorem assures the restriction $(q,\lambda )=W\mathit{x}+U(W\mathit{x},\beta )$, where $U\left(W\mathit{x}\right)\in \mathrm{range}\left(d^2\mathcal{L}\left(q^{*}\right)\right)$, and $W\mathit{x}\in {\mathcal{K}}^{*}$ instead of $W\mathit{x}\in ker{d}^2\mathcal{L}\left(q^{*}\right)$ as in (19) [23]. Simply ignoring the space ${\mathcal{K}}_p$ by considering $U\in \mathrm{range}\left(d^2\mathcal{L}\left(q^{*}\right)\right)$ and $W\mathit{x}\in {\mathcal{K}}^{*}$ amounts to setting $W\mathit{x}=k^{*}+k_p$ and $k_p=0$. Since $W\mathit{x}+U$ is still embedded in the larger ${\Re }^{NK+K}$, which contains ${\mathcal{K}}_p$, then derivatives are affected by the implicit $k_p=0$ constraint. This constraint $P_{{\mathcal{K}}_p}(q,\lambda )=k^{*}+U$ is nonlinear (and may not even be tractable) since ${\mathcal{K}}_p$ depends on q, where $P_{{\mathcal{K}}_p}$ is a projection matrix that depends on q (see Theorem 7).

2.6. Isotropy Subgroups $S_m\times S_n$ of $S_N$

The decomposition $ker{d}^2\mathcal{L}\left(q^{*}\right)={\mathcal{K}}_p\times {\mathcal{K}}^{*}$ shows that $\mathrm{Fix}(S_m\times S_n)\cap ker{d}^2\mathcal{L}\left(q^{*}\right)$ is two-dimensional with basis vectors

$$\{{(n{\mathit{y}}^T,\dots ,n{\mathit{y}}^T,-m{\mathit{y}}^T,\dots ,-m{\mathit{y}}^T)}^T,{(n{\mathit{v}}^T,\dots ,n{\mathit{v}}^T,-m{\mathit{v}}^T,\dots ,-m{\mathit{v}}^T)}^T\}.$$

Restricted to ${\mathcal{K}}^{*}$, these isotropy subgroups $S_m\times S_n$ of $S_M$ have one-dimensional fixed point spaces. This assures that we can use Theorem 1. We have the following Lemma.

Lemma 1.

Let $M=m+n$ such that $M>1$ and $m,n>0$. Let ${\mathcal{U}}_m$ be a set of m classes, and let ${\mathcal{U}}_n$ be a set of n classes such that ${\mathcal{U}}_m\cap {\mathcal{U}}_n=\varnothing$ and ${\mathcal{U}}_m\cup {\mathcal{U}}_n=\{1,\cdots ,M\}$. Now define ${\widehat{\mathit{u}}}_{(m,n)}\in {\Re }^{NK}$ such that

$$\begin{array}{c}\hfill {\widehat{\mathit{u}}}_{(m,n)}^i=\left\{\begin{array}{cc}n\mathit{v}& if i\in {\mathcal{U}}_m\hfill \\ -m\mathit{v}& if i\in {\mathcal{U}}_n\hfill \\ \mathbf{0}& otherwise\hfill \end{array}\right.\end{array}$$

where $\mathit{v}$ is defined as in Definition 1.2, and let

$$\begin{array}{c}\hfill {\mathit{u}}_{(m,n)}=\left(\begin{array}{c}{\widehat{\mathit{u}}}_{(m,n)}\\ \mathbf{0}\end{array}\right)\end{array}$$

(26)

where $\mathbf{0}\in {\mathbf{R}}^K$. Then the isotropy subgroup of ${\mathit{u}}_{(m,n)}$ is ${\mathsf{\Sigma }}_{(m,n)}\subset {\mathsf{\Gamma }}_{\mathcal{U}}$ such that ${\mathsf{\Sigma }}_{(m,n)}\cong S_m\times S_n$, where $S_m$ permutes ${\mathit{u}}^i$ when $i\in {\mathcal{U}}_m$, and $S_n$ permutes ${\mathit{u}}^i$ when $i\in {\mathcal{U}}_n$. The fixed point space of ${\mathsf{\Sigma }}_{(m,n)}$ restricted to ${\mathcal{K}}^{*}\subset d^2\mathcal{L}\left(q^{*}\right)$ is one dimensional.

2.7. Bifurcating Branches

Theorem 3.

Let $(q^{*},{\lambda }^{*},{\beta }^{*})$ be an equilibrium of (8) such that $q^{*}$ is M-singular for $1<M\le N$, and the crossing condition

$$d_{\beta }{d}^2\mathcal{L}[\mathit{u},\mathit{u}]-d^3\mathcal{L}[\mathit{u},\mathit{u},L^{-}{d}_{\beta }\nabla \mathcal{L}]\ne 0$$

is satisfied. Then there exists bifurcating solutions, $\left(\begin{array}{c}{q}^{*}\\ {\lambda }^{*}\\ {\beta }^{*}\end{array}\right)+\left(\begin{array}{c}t{\mathit{u}}_{(m,n)}\\ \beta \left(t\right)\end{array}\right)$, where ${\mathit{u}}_{(m,n)}\in {\mathcal{K}}^{*}$ is defined in (26), for every pair $(m,n)$ such that $M=m+n$, each with an isotropy group isomorphic to $S_m\times S_n$.

Proof. 

We mimic the proof of the Equivariant Branching Lemma. Let $\mathit{u}:={\mathit{u}}_{(m,n)}\in \mathrm{Fix}(S_m\times S_n)\cap {\mathcal{K}}^{*}$ and let V be a matrix with columns composed of the $M-1$ vectors $\left\{V_i\right\}$. Thus, there exists ${\mathit{x}}_0\in {\Re }^{M-1}$ so that $\mathit{u}=V{\mathit{x}}_0$. Since $r(\mathrm{Fix}(S_m\times S_n)\cap {\mathcal{K}}^{*})\subseteq \mathrm{Fix}(S_m\times S_n)\cap {\mathcal{K}}^{*}$ (for every $\sigma \in S_m\times S_n$, $r\left(V\mathit{x}\right)=r\left(\sigma V\mathit{x}\right)$ ($\mathit{u}\in \mathrm{Fix}(S_M\times S_n$) that equals $\sigma r\left(V\mathit{x}\right)$ (by equivariance)), then $r(t{\mathit{x}}_0,\beta )=h(t,\beta ){\mathit{x}}_0$, where r is the Liapunov–Schmidt reduction (19), and h is a polynomial in t.

Since ${\mathcal{K}}^{*}$ is $S_M$-irreducible, then $\mathrm{Fix}\left(S_M\right)\cap {\mathcal{K}}^{*}=\left\{\mathbf{0}\right\}$ (otherwise, $\sigma \mathit{x}=\mathit{x}$ for some $\mathit{x}\in {\mathcal{K}}^{*}$ for every $\sigma \in S_M$, which implies that $\mathrm{span}\left(\mathit{x}\right)$ is an invariant subspace of ${\mathcal{K}}^{*}$). Now [22] p. 75 shows that $r(\mathbf{0},\beta )=\mathbf{0}$, and so $h(0,\beta )=0$, from which it follows that $h(t,\beta )=tk(t,\beta )$. Thus,

$$\begin{array}{c}\hfill r(t{\mathit{x}}_0,\beta )=tk(t,\beta ){\mathit{x}}_0.\end{array}$$

(27)

Differentiating with respect to t yields

$$\begin{array}{c}\hfill d_{\mathit{x}}r(t{\mathit{x}}_0,\beta ){\mathit{x}}_0=(k(t,\beta )+t{d}_{t}k(t,\beta )){\mathit{x}}_0,\end{array}$$

(28)

from which it follows that

$$k(t,\beta ){\mathit{x}}_0=d_{\mathit{x}}r(t{\mathit{x}}_0,\beta ){\mathit{x}}_0-t{d}_{t}k(t,\beta ){\mathit{x}}_0,$$

and so $k(\mathbf{0},0)=0$. Furthermore, we see that $d_{\beta }k(0,0){\mathit{x}}_0=d_{\mathit{x},\beta }^{2}r(0,0){\mathit{x}}_0\ne \mathbf{0}$ by assumption (see (23)). This shows that $d_{\beta }k(0,0)$ is a non-zero eigenvalue of $d_{\mathit{x}}r(t{\mathit{x}}_0,\beta )$ with associated eigenvector ${\mathit{x}}_0$. By the Implicit Function Theorem, $k(t,\beta )=0$ has a non-zero unique solution for $\beta =\beta \left(t\right)$. □

2.8. The Crossing Condition for Annealing Problemsn

We next determine how to check the crossing condition in Theorem 3 when F is an annealing problem, as in (2)

$$F(q,\beta )=H\left(q\right)+\beta D\left(q\right).$$

First, we show that the crossing condition can be checked in terms of the Hessian of the function D. Furthermore, when G is strictly concave on $\mathrm{span}\left(\left\{{\mathit{v}}_i\right\}\right)$, then the crossing condition is always satisfied, and every singularity is a bifurcation.

Theorem 4.

The crossing condition

$$d_{\beta }{d}^2\mathcal{L}[\mathit{u},\mathit{u}]-d^3\mathcal{L}[\mathit{u},\mathit{u},L^{-}{d}_{\beta }\nabla \mathcal{L}]\ne 0$$

given in Theorem 3 is satisfied for M-singular q for $M>1$ if $d^{2}D\left(q\right)$ is either positive or negative definite on $\mathrm{span}\left(\left\{{\mathit{v}}_i\right\}\right)$.

Proof. 

Let ${\mathit{x}}_0\in {\Re }^{2M-2}$ so that $\mathit{u}=W{\mathit{x}}_0\in \mathrm{Fix}(S_m\times S_n)\cap {\mathcal{K}}^{*}.$ Multiplying Equation (21) on the left by ${\mathit{x}}_0^T$ and on the right by ${\mathit{x}}_0$ yields

$$\begin{array}{c}\hfill {\mathit{x}}_0^T{d}_{\mathit{x}}r(\mathbf{0},\beta ){\mathit{x}}_0={\mathit{u}}^T{d}_{q,\lambda }^2\mathcal{L}(q^{*},{\lambda }^{*},\beta +{\beta }^{*})(I_{NK+K}+d_{\mathit{w}}U(\mathbf{0},\beta ))\mathit{u}.\end{array}$$

(29)

By Theorem 2, an arbitrary $\mathit{u}\in {\mathcal{K}}^{*}$ can be written as $\mathit{u}=\left(\begin{array}{c}\widehat{\mathit{u}}\\ \mathbf{0}\end{array}\right)$, where $\widehat{\mathit{u}}\in \mathrm{span}\left(\left\{{\mathit{v}}_i\right\}\right)\subset ker{d}^{2}F(q^{*},{\beta }^{*})$. Substituting this into (29) and observing that $d^{2}F(q^{*},\beta +{\beta }^{*})=d^{2}G\left(q^{*}\right)+(\beta +{\beta }^{*})d^{2}D\left(q^{*}\right)=d^{2}F(q^{*},{\beta }^{*})+\beta d^{2}D\left(q^{*}\right)$ yields

$$\begin{array}{c}\hfill {\mathit{x}}_0^T{d}_{\mathit{x}}r(\mathbf{0},\beta ){\mathit{x}}_0=\beta \left(\begin{array}{cc}{\widehat{\mathit{u}}}^T{d}^{2}D\left(q^{*}\right)& {\mathbf{0}}^T\end{array}\right)\left(I_{NK+K}+{\partial }_{\mathit{w}}U(\mathbf{0},\beta )\right)\left(\begin{array}{c}\widehat{\mathit{u}}\\ \mathbf{0}\end{array}\right).\end{array}$$

Differentiating with respect to $\beta$, evaluating at $\beta =0$, and using (20) yields

$$\begin{array}{c}\hfill {\mathit{x}}_0^T{d}_{\mathit{x},\beta }^{2}r(\mathbf{0},0){\mathit{x}}_0={\widehat{\mathit{u}}}^T{d}^{2}D\left(q^{*}\right)\widehat{\mathit{u}},\end{array}$$

(30)

which must be non-zero since we assume that $d^{2}D\left(q\right)$ is either positive or negative definite on $\mathrm{span}\left(\left\{{\mathit{v}}_i\right\}\right)$. □

From (30), we can get an expression for $\xi$, the eigenvalue of $d_{\mathit{x},\beta }^{2}r(0,0)$ with eigenvector ${\mathit{x}}_0$. Substituting $d_{\mathit{x},\beta }^{2}r(\mathbf{0},0){\mathit{x}}_0=\xi {\mathit{x}}_0$ and observing that ${\mathit{x}}_0^T{\mathit{x}}_0={\mathit{x}}_0^T{W}^{T}W{\mathit{x}}_0={\widehat{\mathit{u}}}^T\widehat{\mathit{u}}$ yields

$$\begin{array}{c}\hfill \xi =\frac{{\widehat{\mathit{u}}}^T{d}^{2}D\left(q^{*}\right)\widehat{\mathit{u}}}{\left|\right|\widehat{\mathit{u}}{\left|\right|}^2}.\end{array}$$

(31)

The requirement that $d^{2}D\left(q\right)$ is either positive or negative definite on $\mathrm{span}\left(\left\{{\mathit{v}}_i\right\}\right)$ holds when $d^{2}G\left(q^{*}\right)$ is either negative or positive definite, respectively, on $\mathrm{span}\left(\left\{{\mathit{v}}_i\right\}\right)$.

Lemma 2.

Let $d^{2}F(q^{*},{\beta }^{*}\ne 0)$ be singular where $q^{*}$ is M-singular such that $d^{2}G\left(q^{*}\right)$ is negative (or positive) definite on $\mathrm{span}\left(\left\{{\mathit{v}}_i\right\}\right)$. Then $d^{2}D\left(q^{*}\right)$ is positive (or negative) definite on $\mathrm{span}\left(\left\{{\mathit{v}}_i\right\}\right)$.

Proof. 

If $\mathit{u}\in \mathrm{span}\left(\left\{V_i\right\}\right)\subset ker{d}^{2}F\left(q^{*}\right)$, then ${\mathit{u}}^T{d}^{2}G\left(q^{*}\right)\mathit{u}+{\beta }^{*}{\mathit{u}}^T{d}^{2}D\left(q^{*}\right)\mathit{u}=0.$ Since ${\mathit{u}}^T{d}^{2}G\left(q^{*}\right)\mathit{u}<0$, then ${\mathit{u}}^T{d}^{2}D\left(q^{*}\right)\mathit{u}>0$. □

These results are important for the Information Bottleneck problem (2), where $d^{2}G\left(q\right)=-d^{2}I(Y;Z)$ is only non-positive definite on $ker{d}^{2}F\left(q^{*}\right)$, but is negative definite on $\mathrm{span}\left(\left\{{\mathit{v}}_i\right\}\right)$. Thus, every singularity of the Information Bottleneck with $ker{d}^2\mathcal{L}\left(q^{*}\right)={\mathcal{K}}^{*}\times {\mathcal{K}}_p$ is a bifurcation point. The space ${\mathcal{K}}_p$ does not contain bifurcating branches since the crossing condition is never satisfied there: for $\mathit{u}\in {\mathcal{K}}_p$, ${\widehat{\mathit{u}}}^T{d}^{2}G\left(q\right)\widehat{\mathit{u}}+\beta {\widehat{\mathit{u}}}^T{d}^{2}D\left(q\right)\widehat{\mathit{u}}=0+0$ (by Lemma 42 in [25]), and so (Theorem 109, [25]) $\xi =\frac{{\widehat{\mathit{u}}}^T{d}^{2}D\left(q\right)\widehat{\mathit{u}}}{\parallel \widehat{\mathit{u}}\parallel }=0$.

2.9. Bifurcation Type

Suppose that a bifurcation occurs at $(q^{*},{\lambda }^{*},{\beta }^{*})$, where $q^{*}$ is M-singular. This section examines the type of bifurcation from which emanate the branches

$$\begin{array}{c}\hfill \left(\left(\begin{array}{c}{q}^{*}\\ {\lambda }^{*}\end{array}\right)+t\mathit{u},{\beta }^{*}+\beta \left(t\right)\right),\end{array}$$

whose existence is guaranteed by Theorem 3.

As we showed in [21], the derivative ${\beta }^{\prime }\left(0\right)\ne 0$ indicates a transcritical bifurcation. If ${\beta }^{\prime }\left(0\right)=0$, then the bifurcation is degenerate, and if ${\beta }^{″}\left(0\right)\ne 0$, then we have a pitchfork-like bifurcation. Further, $t{\beta }^{\prime }\left(t\right)<0$ for small t indicates a subcritical bifurcating branch, and $t{\beta }^{\prime }\left(t\right)>0$ for small t indicates a supercritical bifurcating branch.

Expressions for ${\beta }^{\prime }\left(0\right)$ and ${\beta }^{″}\left(0\right)$ are derived as follows. Differentiating $k(t,\beta )=0$ from (27) yields

$$\begin{array}{c}\hfill d_{t}k(t,\beta \left(t\right))+d_{\beta }k(t,\beta \left(t\right)){\beta }^{\prime }\left(t\right)=0,\end{array}$$

(32)

so that ${\beta }^{\prime }\left(t\right)=-\frac{d_{t}k(t,\beta \left(t\right))}{d_{\beta }k(t,\beta \left(t\right))}.$ Differentiating (28) with respect to t and then evaluating at $t=0$ shows that

$$\begin{array}{c}\hfill {\beta }^{\prime }\left(0\right)=\frac{-d_{\mathit{x}}^{2}r(\mathbf{0},0)[{\mathit{x}}_0,{\mathit{x}}_0,{\mathit{x}}_0]}{2\left|\right|{\mathit{x}}_0{\left|\right|}^2\xi }\end{array}$$

(33)

where $d_{\mathit{x}}^{2}r(\mathbf{0},0)[{\mathit{x}}_0,{\mathit{x}}_0,{\mathit{x}}_0]={\sum }_{i,j,k}\frac{{\partial }^{2}r}{\partial {\left[\mathit{x}\right]}_i\partial {\left[\mathit{x}\right]}_j\partial {\left[\mathit{x}\right]}_k}(\mathbf{0},0){\left[{\mathit{x}}_0\right]}_i{\left[{\mathit{x}}_0\right]}_j{\left[{\mathit{x}}_0\right]}_k$ (see (24)). As shown in the proof to Theorem 3, $\xi =d_{\beta }k(\mathbf{0},0)$ is the non-zero eigenvalue of $d_{\mathit{x},\beta }^{2}r(0,0)$ with eigenvector ${\mathit{x}}_0$.

This expression is similar to the one given in [22] p. 90. The numerator can be calculated via (24). In [21], we showed that ${\beta }^{\prime }\left(0\right)=0$. We have the same result in the present case.

Theorem 5.

If $q^{*}$ is M-singular for $1<M\le N$, then all of the bifurcating branches guaranteed by Theorem 3 are degenerate, i.e., ${\beta }^{\prime }\left(0\right)=0$.

Proof. 

To show that the numerator of (33) $d_{\mathit{x}}^{2}r(\mathbf{0},0)=\mathbf{0}$, expand $r_i$, the ith component of r, about $\mathit{x}=\mathbf{0}$,

$$\begin{array}{ccc}\hfill r_i(\mathit{x},\beta )& =& r_i(\mathbf{0},\beta )+d_{\mathit{x}}{r}_i{(\mathbf{0},\beta )}^T\mathit{x}+{\mathit{x}}^T{d}_{\mathit{x}}^2{r}_i(0,\beta )\mathit{x}+\mathcal{O}\left({\mathit{x}}^3\right)\hfill \\ & =& d_{\mathit{x}}{r}_i{(\mathbf{0},\beta )}^T\mathit{x}+{\mathit{x}}^T{d}_{\mathit{x}}^2{r}_i(0,\beta )\mathit{x}+\mathcal{O}\left({\mathit{x}}^3\right),\hfill \end{array}$$

and so

$$\begin{array}{ccc}\hfill r_i(\mathit{x},0)& =& {\mathit{x}}^T{d}_{\mathit{x}}^2{r}_i(0,0)\mathit{x}+\mathcal{O}\left({\mathit{x}}^3\right).\hfill \end{array}$$

Applying the equivariance relation $Ar(\mathit{x},0)=r(A\mathit{x},0)$, where A is any element of the group isomorphic to $S_M$ that acts on r in ${\mathbf{R}}^{M-1}$, and equating the quadratic terms yields

$$A\left(\begin{array}{c}{\mathit{x}}^T{d}_{\mathit{x}}^2{r}_1\mathit{x}\\ {\mathit{x}}^T{d}_{\mathit{x}}^2{r}_2\mathit{x}\\ ⋮\\ {\mathit{x}}^T{d}_{\mathit{x}}^2{r}_{M-1}\mathit{x}\end{array}\right)=\left(\begin{array}{c}{\mathit{x}}^T{A}^T{d}_{\mathit{x}}^2{r}_{1}A\mathit{x}\\ {\mathit{x}}^T{A}^T{d}_{\mathit{x}}^2{r}_{2}A\mathit{x}\\ ⋮\\ {\mathit{x}}^T{A}^T{d}_{\mathit{x}}^2{r}_{M-1}A\mathit{x}\end{array}\right).$$

By (24), the diagonal $\frac{{\partial }^2{r}_i}{\partial x_i\partial x_i}(\mathbf{0},0)=0$ for each i as well as for all of the “multi-diagonals”. This shows that $\frac{{\partial }^2{r}_i}{\partial x_j\partial x_k}(\mathbf{0},0)=0$ for every $i,j,k$ (see Theorem 124 in [25]). □

When ${\beta }^{\prime }\left(0\right)=0$, we need to compute ${\beta }^{″}\left(0\right)$ to determine whether a branch is subcritical or supercritical. Differentiating (32) and setting $t=0$ shows that ${\beta }^{″}\left(0\right)=-\frac{d_t^{2}k(0,0)}{d_{\beta }k(\mathbf{0},0)}.$ Differentiating (28) twice and solving for $d_t^{2}k(0,0)$ shows that

$$\begin{array}{c}\hfill {\beta }^{″}\left(0\right)=\frac{-d_{\mathit{x}}^{3}r(\mathbf{0},0)[{\mathit{x}}_0,{\mathit{x}}_0,{\mathit{x}}_0,{\mathit{x}}_0]}{3\left|\right|{\mathit{x}}_0{\left|\right|}^2\xi }\end{array}$$

(34)

where $W{\mathit{x}}_0=\mathit{u}={\mathit{u}}_{(m,n)}$. Use Equation (25) to calculate the numerator, and $\xi =d_{\beta }k(\mathbf{0},0)$ is the non-zero eigenvalue of $d_{\mathit{x},\beta }^{2}r(0,0)$ with eigenvector ${\mathit{x}}_0$, for which we give an explicit expression in (31) when F is an annealing problem.

If ${\beta }^{″}\left(0\right)\ne 0$, which we expect to be true generically, then Theorem 5 shows that the bifurcation guaranteed by Theorem 3 is pitchfork-like.

2.10. Stability and Optimality

The next Theorem relates the stability of equilibria $(q^{*},{\lambda }^{*},\beta )$ in the flow (8) with optimality of $q^{*}$ in Problem (1). In particular, if a bifurcating branch corresponds to an eigenvalue of $d^2\mathcal{L}\left(q^{*}\right)$ changing from negative to positive, then the branch consists of stationary points $(q^{*},{\beta }^{*})$ that are not solutions of (1). Positive eigenvalues of $d^2\mathcal{L}\left(q^{*}\right)$ do not necessarily show that $q^{*}$ is not a solution of (1) (see Remark 1). For example, see page 668 of [21]. A proof of this theorem is given in [21].

Theorem 6.

For each bifurcating branch guaranteed by Theorem 3, $\mathit{u}$ is an eigenvector of $d^2\mathcal{L}(\left(\begin{array}{c}{q}^{*}\\ {\lambda }^{*}\end{array}\right)+t\mathit{u},{\beta }^{*}+\beta \left(t\right))$ for sufficiently small t. Furthermore, if the corresponding eigenvalue is positive, then the branch consists of unstable stationary points that are not solutions to (1).

2.11. Structure of the Symmetry Projection

The matrix $P_R\left(q^{*}\right)$ that projects $(q,\lambda )\in {\Re }^{NK+K}$ onto $\mathrm{range}\left(d^2\mathcal{L}\left(q^{*}\right)\right)\times {\mathcal{K}}^{*}$ by annihilating ${\mathcal{K}}_p$ is important for numerical computations for equilibria of IB, since we may want to take each equilibrium found by Newton’s method and take out any part in ${\mathcal{K}}_p$. $P_R$ is written as a function of q since its constitutive vectors $\mathit{y}$ (from Definition 1) depend on q. The following theorems clarify the structure of this projection.

Theorem 7.

$P_R\left(q\right)=I-P_{{\mathcal{K}}_p}\left(q\right)$, where $P_{{\mathcal{K}}_p}=\left(\begin{array}{cc}A& \mathbf{0}\\ \mathbf{0}& \mathbf{0}\end{array}\right).$$P_R$ and $P_{{\mathcal{K}}_p}$ are $(NK+K)\times (NK+K)$. The matrix A is $NK\times NK$ with $N^2$ blocks, ${\left\{A_{ij}\right\}}_{i,j=1}^N$, of size $K\times K$, defined by

$$\begin{array}{c}\hfill A_{i,j}=\left\{\begin{array}{cc}(M-1)\mathit{y}{\mathit{y}}^T& if 1\le i=j\le M\\ -\mathit{y}{\mathit{y}}^T& if 1\le i\ne j\le M\\ \mathbf{0}& otherwise\end{array}\right.\end{array}$$

For example, if $M=N=3$, then

$$\begin{array}{c}\hfill P_R=I-\left(\begin{array}{cccc}2\mathit{y}{\mathit{y}}^T& -\mathit{y}{\mathit{y}}^T& -\mathit{y}{\mathit{y}}^T& \mathbf{0}\\ -\mathit{y}{\mathit{y}}^T& 2\mathit{y}{\mathit{y}}^T& -\mathit{y}{\mathit{y}}^T& \mathbf{0}\\ -\mathit{y}{\mathit{y}}^T& -\mathit{y}{\mathit{y}}^T& 2\mathit{y}{\mathit{y}}^T& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}\end{array}\right)=I-\left(\begin{array}{cccc}(N-1)& -1& -1& 0\\ -1& (N-1)& -1& 0\\ -1& -1& (N-1)& 0\\ 0& 0& 0& 0\end{array}\right)\otimes \mathit{y}{\mathit{y}}^T.\end{array}$$

Proof. 

Theorem 2 gives the basis of ${\mathcal{K}}_p$ as ${\left\{Y_i\right\}}_{i=1}^{M-1}$. Let Y be the $(NK+K)\times (M-1)$ matrix whose columns are the vectors $\left\{Y_i\right\}$. For example, if $M=3$ and $N=4$, then $Y=\left(\begin{array}{cc}\mathit{y}& \mathbf{0}\\ \mathbf{0}& \mathit{y}\\ -\mathit{y}& -\mathit{y}\\ \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}\end{array}\right)$. Thus, the matrix that projects onto ${\mathcal{K}}_p$ is $P_{{\mathcal{K}}_p}=Y{\left(Y^{T}Y\right)}^{-1}{Y}^T$, and the projection matrix onto $\mathrm{range}\left(d^2\mathcal{L}\left(q^{*}\right)\right)$ is $P_R=I-P_{{\mathcal{K}}_p}$. Direct multiplication of $Y{\left(Y^{T}Y\right)}^{-1}{Y}^T$, with an appeal to Lemma 34 in [25] to compute the inverse, shows that $P_{{\mathcal{K}}_p}=\frac{1}{N{\mathit{y}}^T\mathit{y}}\left(\begin{array}{cc}A& \mathbf{0}\\ \mathbf{0}& \mathbf{0}\end{array}\right)$. Dropping the constant yields the result. □

For the Information Bottleneck, the matrix $P_R$ is easy to calculate, since $\mathit{y}=q^i$ for any $i\in \mathcal{U}$. For example, when $q=q_{\frac{1}{N}}$, then ${\mathit{y}}^T\mathit{y}=\frac{K}{N^2}$ and $\mathit{y}{\mathit{y}}^T=\frac{1}{N^2}\mathbf{1}$, and so

$$P_{{\mathcal{K}}_p}=\frac{1}{NK}\left(\begin{array}{ccccc}(N-1)& -1& \cdots & -1& 0\\ -1& (N-1)& \cdots & -1& 0\\ ⋮& ⋮& & ⋮& ⋮\\ -1& -1& \cdots & (N-1)& 0\\ 0& 0& 0& 0& 0\end{array}\right)\otimes \mathbf{1}$$

where 1 is a $K\times K$ matrix of 1s. Thus,

$$P_R=I_{NK+K}-\left(\begin{array}{ccccc}(N-1)& -1& \cdots & -1& 0\\ -1& (N-1)& \cdots & -1& 0\\ ⋮& ⋮& & ⋮& ⋮\\ -1& -1& \cdots & (N-1)& 0\\ 0& 0& 0& 0& 0\end{array}\right)\otimes \mathbf{1}.$$

Theorem 8.

The symmetry group $S_M$ commutes with the matrix $P_R$, which projects onto ${\Re }^{NK+K}\setminus {\mathcal{K}}_p$.

Proof. 

Let $P:=P_R$ be the matrix that projects onto $\mathrm{range}{d}^2\mathcal{L}\left(q^{*}\right)\times {\mathcal{K}}^{*}={\Re }^{NK+K}\setminus {\mathcal{K}}_p$. Since ${\Re }^{NK+K}=\mathrm{range}{d}^2\mathcal{L}\left(q^{*}\right)\times {\mathcal{K}}^{*}\times {\mathcal{K}}_p$, then any $x\in {\Re }^{NK+K}$ can be decomposed in the respective subspaces as $x=r+k^{*}+k_p$. Let $\sigma$ be an arbitrary permutation matrix in $S_M$. Then $\sigma P\left(\begin{array}{c}q\\ \lambda \end{array}\right)=\sigma P(r+k^{*}+k_p)=\sigma (r+v)$. Since $\mathrm{range}{d}^2\mathcal{L}\left(q^{*}\right)$, ${\mathcal{K}}^{*}$ and ${\mathcal{K}}_p$ are all $S_M$ invariant; then $\sigma (r+v)\in \mathrm{range}{d}^2\mathcal{L}\left(q^{*}\right)\times {\mathcal{K}}^{*}$ implies that $\sigma (r+v)=P\sigma (r+v)$, and $\sigma z\in {\mathcal{K}}_p$ implies that $P\sigma (r+v)=P\sigma (r+v+z).$ Thus, $\sigma Px=P\sigma x$. □

2.12. Visualizations of Sample Resultsn

We illustrate these structures numerically. In [7], we introduced the toy “Four-blob” probability distribution $p(x,y)$ shown in Figure 1.

Figure 1. The probability distribution $p(x,y)$ for the “Four-blob” toy problem for a system of interest $X\to Y$. We use this probability to illustrate some results of the bifurcation analysis reported here.

For the Information Distortion problem (3) [7,12,13] and the synthetic dataset composed of a mixture of four Gaussians (Figure 1), we determined the bifurcation structure of solutions to (3) by annealing in $\beta$ and finding the corresponding stationary points to (1). A typical run of the derived gradient dynamical system tends to follow the main bifurcation branch $S_K\to S_{K-1}$ from the fully symmetric uniform quantizer $q_{\frac{1}{N}}$ ($N=4$ here) to the fully resolved deterministic quantizer (hard clustering) seen at the end in Figure 2. The permutation symmetry is also obvious there—the value of the cost function does not change if the classes along the vertical axis in T are permuted/relabeled. The uniform quantizer $q_{\frac{1}{N}}$ (Item 1 in the figure) plays a special role in the formulation (3), as it is the unique solution to the problem for $\beta =0$ as the maximum entropy solution of ${max}_{q}H\left(T\right|Y)$. Its loss of stability at the first bifurcation for increasing $\beta$ can hence be determined analytically and the first bifurcation structure characterized completely. Because of the “perpetual kernel” of the cost function in (4), the uniform quantizer is just one of a continuous set of “uninformative” quantizers for the IB problem (4): all $\left\{q\right(t\left|y\right):q\left(t\right|y)=f(t\left)\right\}$, having constant probability of assignment of each y to class t, but the assignment weight can be different for different classes. Such a structure does not change the value of the cost function in the IB problem (4) (but does change it for (3), which hence does not have this degeneracy). We address the degeneracy of the IB optimization by projecting onto the subspace that has the correct symmetry (i.e., just the uniform quantizer $q_{\frac{1}{N}}$ in this case), as outlined in Remark 2.

Figure 2. The bifurcations of the solutions $(q^{*},\beta )$ to the Information Distortion problem (3). For the mixture of 4 well-separated Gaussians shown in Figure 1, the behavior of $D\left(q\right)=I(X;T)$ as a function of $\beta$ is shown in the top panel, and some of the solutions $q^{*}\left(T\right|Y)$ are shown in the bottom panels. Item 1 shows the uniform quantizer $q_{\frac{1}{N}}$, assigning equal probability of each $y\in Y$ to belong to one of the four clusters in T. Subsequent items 2–5 point to a set of partially resolved quantizations, in which subsets of Y are assigned with high probability to one (2) or more (3–5) classes (dark colors, close to 1), while other subsets are still unresolved (gray levels), albeit as a higher probability than $q_{\frac{1}{N}}$ (darker gray, as some of the classes are excluded after being resolved for another subset). Item 6 shows an almost fully resolved quantizer at sufficiently high $\beta$. They become fully resolved (deterministic; $q\left(t\right|y)=1$ or 0) as $\beta \to \infty$ (not shown).

A more-thorough structure of the bifurcation diagram, using the analysis presented above, is shown in Figure 3.

Figure 3. (A) The bifurcation structure of stationary points of the Information Distortion problem (3), a problem of form (2). We found these points by annealing in $\beta$ and finding stationary points for Problem (1) using the algorithm presented in [28]. A square indicates where a bifurcation occurs. (B) A close-up of the subcritical bifurcation at $\beta \approx 1.038706$, indicated by a square. Observe the subcritical bifurcating branch, and the subsequent saddle-node bifurcation at $\beta \approx 1.037479$, indicated by another square. We applied Theorem 6 to show that the subcritical bifurcating branch is composed of quantizers that are solutions of (3) but not of (1).

Similar to the results we presented in [28], the close-up of the bifurcation at $\beta \approx 1.038706$ in Figure 3B shows a subcritical bifurcating branch (a first-order phase transition) that consists of stationary points of Problem (1). By projecting the Hessian ${\Delta }_q(G\left(q^{*}\right)+\beta D\left(q^{*}\right))$ onto each of the kernels referenced in Theorem 6, we determined that the points on this subcritical branch are not solutions of (1), and yet they are solutions of (2).

Furthermore, observe that Figure 3B indicates that a saddle-node bifurcation occurs at $\beta \approx 1.037479$. That this is indeed the case was proved in [21]. In fact, for any problem of the form (2), these are the only two types of bifurcations to be expected: pitchfork and saddle-node.

3. Conclusions and Discussion

The main goal of this contribution was to show that information-based distortion-annealing problems such as (2) have an interesting mathematical structure. The most interesting aspects of that mathematical structure are driven by the symmetries present in the cost functions—their invariance to actions of the permutation group $S_N$, represented as relabeling of the reproduction classes. Such a structure would hold for any biclustering problem [4] that relies on the intrinsic interaction of a pair of variables for unsupervised clustering. The second mathematical structure that we used successfully was bifurcation theory, which allowed us to identify and study the discrete points at which the character of the cost function changed. The combination of those two tools in [20] allowed us to explicitly compute the value of the annealing parameter $\beta$ at which the initial maximum at the uniform quantizer $q_{\frac{1}{N}}$ of (1) loses stability. We concluded that for a fixed system $C\to Y$ characterized by $p(X,Y)$, this value is the same for both problems, that it does not depend on the number of elements of the reproduction variable T, and that it is always greater than 1. We further introduced an eigenvalue problem that links the critical values of $\beta$ and q for bifurcations, or phase transitions, branching off arbitrary intermediate solutions.

Even though the cost functions $F_{IB}$ (4) and $F_H$ (3) have similar properties, they also differ in some important aspects. We have shown that the function $F_{IB}$ is degenerate since its constitutive functions $I(X;Y)$ and $I(X;T)$ are not strictly convex in q. That introduces additional invariances and singularities that are always preserved, which makes phase transitions more difficult to detect (e.g., the ”uninformative quantizers” $q\left(t\right|y)=f(t)$ only) and post-transition directions more difficult to determine. In contrast, $F_H$ is strictly convex except at points of phase transitions. The theory we developed here allows us to identify bifurcation directions and determine their stability. Despite the presence of a high-dimensional null space at bifurcations, the symmetries restrict the allowed transition dimensions to multiple co-dimension 1 transitions, all related by group transformations. We achieved that here with three main results. Theorem 8 extended the Equivariant Branching Lemma 1 to the Information Bottleneck case with additional translation invariance. Theorem 4 identified specific conditions at which a bifurcation of the gradient flow (8) occurs. This condition is computable analytically for the initial bifurcation off the uniform quantizer $q_{\frac{1}{N}}$ and with numeric continuation for subsequent bifurcation. Finally, in Section 2.9, we provided checks for the types of bifurcations that occur, giving conditions to detect saddle-node and pitchfork bifurcations and to determine whether pitchforks are supercritical (second-order phase transitions) or subcritical (leading to first-order phase transitions discontinuous in $\beta$). The combination of the three results, together with our previous results in [20], completely characterize the local bifurcation structure of Information Bottleneck-type problems with or without the added translation symmetry.

Despite the further development of the bifurcation formalism for IB presented her, there are still open questions that this manuscript did not resolve. In particular, we still cannot confirm or reject the conjecture that the set of $S_K$ symmetric soft-clustering branches connected through symmetry-breaking bifurcations leads to the global hard-clustering optima at $\beta \to \infty$ (multiple equivalent solutions connected by the permutation symmetry of the problem). We believe this is partially due to a discrepancy between practical observations and theoretical results. In particular, we and other practitioners [29,30] note that the only observed symmetry-breaking bifurcations during optimization are of the kind $S_M\to S_{M-1}$, while the theory allows for arbitrary $S_M\to S_m\times S_n$ bifurcations. The latter are known to happen and be stable in other biological systems and circumstances [26,31]. This suggests a research approach of comparing and contrasting the different systems that possess the same $S_N$ symmetry and symmetry-breaking bifurcations to lead to breakthroughs in this application to optimization in the Information Bottleneck problem.

An additional open problem involves the use of continuous variables, already noted in [5] and explored further in [32,33]. This approach, while important for many real-world problems, involves the application of additional mathematical tools, namely Calculus of Variations [34], which further increases the complexity of an otherwise already complex problem. These difficulties are illustrated in a pair of papers [35,36] that use the continuous formulation. They do present some significant results on conditions of learnability, but both papers manage to only get bounds on $\beta$ under which learnability (optimal solutions beyond the “uninformative” quantizer) can be achieved. This is possibly due to the presence of continuous spectra in covariance operators of continuous quantizers, something that we avoid by focusing on finite spaces. As a consequence, here and in prior work [20], we show specific values for $\beta$ for the initial bifurcation from the uniform quantizer, which supports nontrivial clustering. We consider formulation with continuous variables beyond the scope of this manuscript, but look forward to the development of additional techniques to incorporate this important case in the bifurcation framework presented here. Regardless of such developments, any practical problem with numeric optimization will involve discretization of the continuous variables, which effectively converts a continuous problem to the discrete state discussed here.