Phase Transition in Ant Colony Optimization

Abstract

Ant colony optimization (ACO) is a stochastic optimization algorithm inspired by the foraging behavior of ants. We investigate a simplified computational model of ACO, wherein ants sequentially engage in binary decision-making tasks, leaving pheromone trails contingent upon their choices. The quantity of pheromone left is the number of correct answers. We scrutinize the impact of a salient parameter in the ACO algorithm, specifically, the exponent $\alpha$, which governs the pheromone levels in the stochastic choice function. In the absence of pheromone evaporation, the system is accurately modeled as a multivariate nonlinear Pólya urn, undergoing phase transition as $\alpha$ varies. The probability of selecting the correct answer for each question asymptotically approaches the stable fixed point of the nonlinear Pólya urn. The system exhibits dual stable fixed points for $\alpha \ge {\alpha }_c$ and a singular stable fixed point for $\alpha <{\alpha }_c$ where ${\alpha }_c$ is the critical value. When pheromone evaporates over a time scale $\tau$, the phase transition does not occur and leads to a bimodal stationary distribution of probabilities for $\alpha \ge {\alpha }_c$ and a monomodal distribution for $\alpha <{\alpha }_c$.

1. Introduction

Sociophysics emerged in the 1970s and has evolved into a captivating research field within statistical physics [1,2]. In particular, herding behavior, or the inclination to follow the majority, has captured the attention of many researchers due to its pivotal role in understanding social phenomena [3,4,5,6,7]. Various probabilistic models have been proposed to describe herding behavior, with one notable example being the ant recruitment model [8,9]. This model explains the intermittent oscillation observed in ants when they are presented with two identical food sources [10,11]. When ants choose a food source from among two food sources, the ant recruitment model incorporates a straightforward herding mechanism in which a randomly selected ant chooses one of the two based on the number of ants that have already made the same choice. Scouts play a crucial role by exploring the terrain to locate food sources [12,13]. When a scout discovers food, it returns to the nest, leaving a pheromone trail in its wake. Other ants are drawn to these pheromone marks and consequently become recruited to forage at the food source.

Ant colony optimization (ACO) is a model-based meta-heuristic inspired by the foraging behavior of ants in their search for the shortest path to food sources [14,15,16,17,18]. While ants may not be highly intelligent individually, they collectively find the shortest path by following pheromone trails left by their fellow ants. The optimal path is determined by the route on which the maximum number of ants travel. Consider a classic problem known as the traveling salesman problem (TSP), which involves finding the shortest possible route that visits each city represented as vertices in a given graph exactly once and returning to the origin city [19]. In ACO, ants make decisions about their next city to visit based on a concept called “pheromone”. Pheromone represents the preference for a particular choice and is collaboratively learned by the ants during their search process [15]. In the context of the TSP defined on a graph, pheromone values are typically associated with edges between cities, reflecting the preference for traveling from one city to another along the corresponding edges. These pheromone values are learned through a reinforcement strategy, where each ant reinforces its chosen paths based on the quality of the solution constructed. This quality is often determined using the inverse of the total length of the route. ACO has found successful applications in various industrial and academic constraint optimization problems and has become one of the most popular methods [20,21,22,23].

ACO has seen significant improvements, and modern ACO algorithms deviate substantially from the original ACO [24]. The fundamental modification lies in controlling the diversity of solutions and achieving convergence [25,26]. In this context, “convergence” refers to the tendency of ants to cluster around similar solutions in the neighborhood and ultimately converge toward the same solution. Early convergence to a small region of the search space leaves large enough sections of the search space unexplored and fails to find suitable solutions. On the other hand, quite slow convergence means that the computational cost required to reach acceptable solutions is high, rendering the search inefficient. Diversity control aims to prevent complete convergence by slowing down the search process.

Many algorithms have been proposed for controlling the diversity of the ACO algorithm. One of the diversity control mechanisms involves modifying the probabilistic decision function [24,27]. Bernd Meyer studied the influence of $\alpha$, the exponent on the pheromone level in the selection function, and suggested that $\alpha$ qualitatively determines diversity and convergence behavior. Additionally, Meyer introduced a dynamic $\alpha$ that changes throughout the search process to enhance search efficiency, a technique known as $\alpha$-“annealing”. Meyer also emphasized the significance of noise in ACO using stochastic differential equations in both static and dynamic environments [24,28,29,30]. Ants respond to a two-choice question, and the noisy communication among ants prevents them from selecting suboptimal choice. In this paper, we study a simple model of ACO in which ants sequentially answer a series of two-choice quizzes. We investigate the phase transition and the qualitative change of the convergence behavior by varying $\alpha$. We start from the discrete time formulation of the problem. Our approach can be straightforwardly generalized to ACO for many-body problems, allowing us to derive the model parameters of the continuous time approach concretely. Additionally, there are various possibilities for the continuous time limit, apart from Wiener noise [31]. In Section 2, we introduce a model and derive stochastic differential equations (SDEs) using diffusion approximation. In Section 3, we investigate the time evolution and examine the effect of $\alpha$ on the convergence properties of the solutions. Section 4 provides a summary of the results. Appendix A explains the estimation of the initial conditions for the SDEs.

2. Method and Definitions

There are N two-choice questions, each of which is answered by quite a large number of ants sequentially [32]. These questions are labeled by $n=1,\dots ,N$. The answer provided by the tth ant is denoted as $X(n,t)\in \{0,1\}$, where $X(n,t)=1$ indicates a correct answer, and $X(n,t)=0$ indicates an incorrect answer. In ACO, multiple ants typically search for the optimal solution simultaneously in each iteration. However, in our model, only one ant conducts the search. Each ant receives 1 point for a correct answer. After ant t has answered all N questions, the total points (TPs) earned by the ant can be calculated using the following equation:

$$\mathrm{TP}\left(t\right)=\sum _{n=1}^{N}X(n,t).$$

Ant t deposits pheromones on their answer $X(n,t)\in \{0,1\}$. The amount of the pheromones is $\mathrm{TP}\left(t\right)$. We assume that the pheromones evaporate and decrease by $e^{-1/\tau }$ per unit time, where $\tau$ represents the time scale of the pheromone evaporation.

Ant $t+1$ observes the total values of the pheromones associated with question m for each choice $X(m,t+1)=\{0,1\}$. We denote the total value of pheromones that remains on all the questions after ant t has answered,

$$S\left(t\right)\equiv \sum _{s=1}^t\mathrm{TP}\left(s\right)e^{-(t-s)/\tau }=\sum _{s=1}^t\sum _{n=1}^{N}X(n,s)e^{-(t-s)/\tau }.$$

(1)

Then, the remaining pheromone on $X(m,s)=1,1\le s\le t$ is

$$S(m,t)\equiv \sum _{s=1}^t\mathrm{TP}\left(s\right)X(m,s)e^{-(t-s)/\tau }=\sum _{s=1}^t\sum _{n=1}^{N}X(n,s)X(m,s)e^{-(t-s)/\tau }.$$

(2)

The remaining pheromone on $X(m,s)=0,1\le s\le t$ is given by $S\left(t\right)-S(m,t)$.

Ants are not highly intelligent, and the probability of them making the correct choice in the two-choice questions by themselves is 1/2. The information provided by TPs gives them an indirect clue about the correct choice. If TP$\left(s\right)>N/2$, the posterior probability for $X(m,s)=1$ is larger than $1/2$. Similarly, if $S(m,t)$ is greater than $S\left(t\right)/2$, the posterior probability for $X(m,t+1)=1$ is greater than $1/2$. In ACO, a decision function with a positive parameters $\alpha$ and K is introduced that uses the values of the pheromones as follows:

$$P(X(m,t+1)=1)=\frac{{(S(m,t)+K)}^{\alpha }}{{(S(m,t)+K)}^{\alpha }+{(S\left(t\right)-S(m,t)+K)}^{\alpha }}.$$

Here, the exponent $\alpha$ determines the response of the choice to the values of the pheromones. As K is positive, one can avoid $S(m,t)=0$ and $S(m,t)=S\left(t\right)$ becoming the absorbing states of the process. However, instead of K, we adopt the following form for the choice function:

$$P(X(m,t+1)=1)=(1-ϵ)\frac{S^{\alpha }(m,t)}{S^{\alpha }(m,t)+{(S\left(t\right)-S(m,t))}^{\alpha }}+\frac{1}{2}ϵ.$$

(3)

Here, $ϵ>0$ is a small positive constant to avoid the absorbing states $S(m,t)=0$ and $S(m,t)=S\left(t\right)$ of the stochastic process [8].

We denote the ratio of the remaining pheromones on the correct choices as $Z(m,t)$,

$$Z(m,t)\equiv \frac{S(m,t)}{S\left(t\right)}.$$

We divide both the denominator and numerator of Equation (3) by $S^{\alpha }\left(t\right)$, and the probability of the correct choice $X(m,t+1)=1$ is expressed as

$$P(X(m,t+1)=1)=(1-ϵ)\left(\frac{Z^{\alpha }(m,t)}{Z^{\alpha }(m,t)+{(1-Z(m,t))}^{\alpha }}\right)+\frac{1}{2}ϵ=f\left(Z(m,t)\right).$$

Here, $f\left(z\right)$ is defined as

$$f\left(z\right)\equiv (1-ϵ)\left(\frac{z^{\alpha }}{z^{\alpha }+{(1-z)}^{\alpha }}\right)+\frac{1}{2}ϵ.$$

(4)

We note that $f(1/2)=1/2$ and $f(1-x)=1-f\left(x\right)$. The slope of $f\left(x\right)$ at $x=1/2$ is $(1-ϵ)\alpha$. We also introduce the discount factor $D\left(t\right)$ and the ratio of correct answers $Z\left(t\right)$ as

$$\begin{array}{ccc}\hfill D\left(t\right)& =& \sum _{n=1}^N\sum _{s=1}^t{e}^{-(t-s)/\tau }=N\left(\frac{1-e^{-t/\tau }}{1-e^{-1/\tau }}\right),\hfill \\ \hfill Z\left(t\right)& =& \frac{S\left(t\right)}{D\left(t\right)}.\hfill \end{array}$$

Stochastic Differential Equations

First, let us derive the recursive relation for $S\left(t\right)$ and $D\left(t\right)$. According to the definition, $D(t+1)$ and $S(t+1)$ obey the next recursive relations

$$\begin{array}{ccc}\hfill D(t+1)& =& N+D\left(t\right)e^{-1/\tau },\hfill \\ \hfill S(t+1)& =& \sum _{i=1}^{N}X(i,t+1)+S\left(t\right)e^{-1/\tau }.\hfill \end{array}$$

If $\tau$ is finite, for $t\gg \tau \gg 1$, we have $D\left(t\right)\simeq N\tau$. In the limit $\tau \to \infty$, the pheromone does not evaporate, and one has $D\left(t\right)=Nt$.

For $N\gg 1$, we can replace the sum of $X(i,s),i=1,\dots ,N$, by the sum of the expected values using the law of large numbers; then:

$$S(t+1)\simeq \sum _{i=1}^{N}f\left(Z(i,t)\right)+S\left(t\right)e^{-1/\tau }.$$

Let us denote the average of $\left\{f\right(Z(i,t)\left)\right\}$ as

$$\overline{f\left(Z\right(i,t\left)\right)}\equiv \frac{1}{N}\sum _{i=1}^{N}f\left(Z(i,t)\right).$$

Then,

$$S(t+1)\simeq S\left(t\right)e^{-1/\tau }+N\overline{f\left(Z\right(i,t\left)\right)}.$$

The recursive relation for $Z\left(t\right)$ is

$$Z(t+1)\simeq \frac{S(t+1)}{D(t+1)}=\frac{D(t+1)-N}{D(t+1)}Z\left(t\right)+\frac{N}{D(t+1)}\overline{f\left(Z\right(i,t)}.$$

$\Delta Z\left(t\right)\equiv Z(t+1)-Z\left(t\right)$ is given as

$$\Delta Z\left(t\right)\simeq \frac{N}{D(t+1)}(\overline{f\left(Z\right(i,t\left)\right)}-Z\left(t\right)).$$

In the continuous time limit $dt=1\to 0$, one obtains:

$$\frac{d}{dt}Z\left(t\right)=\frac{N}{D(t+1)}(\overline{f\left(Z\right(i,t\left)\right)}-Z\left(t\right)).$$

(5)

One sees that $Z\left(t\right)$ converges to $\overline{f\left(Z\right(i,t\left)\right)}$. When the pheromones evaporate and $\tau <\infty$, $D\left(t\right)\simeq N\tau$, and the prefactor of the differential equation is $1/\tau$. If one assumes that the dynamics of $Z(i,t)$ are faster than those of $Z\left(t\right)$ (adiabatic approximation), the time scale of the convergence is given by $\tau$ as $\overline{f\left(Z\right(i,t\left)\right)}-Z\left(t\right)\propto e^{-t/\tau }$. When the pheromone does not evaporate and $\tau \to \infty$, $D\left(t\right)\simeq Nt$. The prefactor of the differential equation is $1/t$, and $\overline{f\left(Z\right(i,t\left)\right)}-Z\left(t\right)\propto t^{-1}$. The convergence becomes quite slow.

Next, let us study the dynamics of $Z(m,t)$. The recursive relation for $S(m,t)$ is

$$S(m,t+1)=S(m,t)e^{-1/\tau }+X(m,t+1)(\sum _{i=1\ne m}^{N}X(i,t+1)+1).$$

$Z(m,t+1)$ is then estimated as

$$\begin{array}{ccc}\hfill Z(m,t+1)& =& \frac{S(m,t+1)}{S(t+1)}=\frac{S\left(t\right)e^{-1/\tau }}{S(t+1)}Z(m,t)+\frac{{\sum }_{i\ne m}X(i,t+1)+1}{S(t+1)}X(m,t+1)\hfill \\ & \simeq & \frac{S(t+1)-N\overline{f\left(Z\right(i,t\left)\right)}}{S(t+1)}Z(m,t)+\frac{N\overline{f\left(Z\right(i,t\left)\right)}+(1-f\left(Z(m,t)\right)}{S(t+1)}X(m,t+1).\hfill \end{array}$$

Using $S(t+1)=D(t+1)Z(t+1)$, one obtains:

$$\Delta Z(m,t)=\frac{N\overline{f\left(Z\right(i,t\left)\right)}}{D(t+1)Z(t+1)}\left[\left(1+\frac{1-f\left(Z\right(m,t\left)\right)}{N\overline{f\left(Z\right(i,t\left)\right)}}\right)X(m,t+1)-Z(m,t)\right].$$

We denote the history of $Z\left(s\right),\left\{Z\right(i,s\left)\right\}$, $s=1,\dots ,t$, as $H_t$, and the conditional expected value of $\Delta Z(m,t)$ is estimated as

$$E(\Delta Z(m,t)|H_t)=\frac{N\overline{f\left(Z\right(i,t\left)\right)}}{D(t+1)Z(t+1)}\left[\left(1+\frac{1-f\left(Z\right(m,t\left)\right)}{N\overline{f\left(Z\right(i,t\left)\right)}}\right)f\left(Z(m,t)\right)-Z(m,t)\right].$$

(6)

Likewise, the conditional variance of $\Delta Z(m,t)$ can be approximated as

$$V(\Delta Z(m,t)|H_t)\simeq {\left(\frac{N\overline{f\left(Z\right(i,t)}}{D(t+1)Z(t+1)}\right)}^{2}f\left(Z(m,t)\right)[1-f\left(Z(m,t)\right)].$$

Here, we neglect the subleading terms in ${\left(1+\frac{1-f\left(Z\right(m,t\left)\right)}{N\overline{f\left(Z\right(i,t\left)\right)}}\right)}^2$. We read the drift and diffusion term from the results and the SDEs are

$$dZ(m,t)=E(\Delta Z(m,t)|H_t)dt+\sqrt{V(\Delta Z(m,t)|H_t)}dW\left(t\right),m=1,\cdots ,N.$$

(7)

Here, $W\left(t\right)$ is the Wiener process. Equations (5) and (7) describe the dynamics of the system. The system can be described as a multivariate Pólya urn process.

We note that $\frac{1-f\left(Z\right(m,t\left)\right)}{N\overline{f\left(Z\right(i,t\left)\right)}}$ in Equation (6) breaks the $Z_2$ symmetry of the system. If one neglects the term, $E(\Delta Z(m,t)|H_t)$ is proportional to $f\left(Z\right(m,t\left)\right)-Z(m,t)$. As $f(1-x)=1-f\left(x\right)$, $f(1-x)-(1-x)=-\left(f\right(x)-x)$ holds. $Z(m,t)$ and $1-Z(m,t)$ obey the same dynamics, and we call the symmetry $Z_2$ symmetry. The term is always positive and drives $Z(m,t)$ in the positive direction. As the term is proportional to $1/N$, the strength of the $Z_2$-symmetry-breaking field becomes smaller as N becomes larger.

3. Results

We analyze the SDEs given in Equation (7) and investigate the convergence properties of $Z(m,t)$. As the convergence behavior relies on the initial value of $Z(m,t=t_0)$ in the context of the nonlinear Pólya urn model, we commence by examining the initial distribution of $Z(m,t)$.

3.1. Initial Distribution of $\left\{Z\right(m,t\left)\right\}$

We assume that ants adopt $\alpha =0$ and do not respond to the values of the pheromones for $t=1,\cdots ,t_0$. The ants answer the questions independently, and $P\left(X\right(i,t)=1)=1/2$. We estimate $Z\left(t\right)$ and $Z(m,t)$ for $\tau <t\le t_0$ as follows:

$$\begin{array}{ccc}\hfill Z\left(t\right)& \sim & N\left(\frac{1}{2},\frac{D_h\left(t\right)}{4D^2\left(t\right)}\right),\hfill \\ \hfill Z(m,t)& \sim & N\left(\frac{1}{2}+\frac{1}{2N},\frac{N{D}_h\left(t\right)}{4D^2\left(t\right)}\right),\hfill \\ \hfill D_h\left(t\right)& \equiv & N\sum _{s=1}^t{e}^{-2(t-s)/\tau }=N·\frac{1-e^{-2t/\tau }}{1-e^{-2/\tau }},\hfill \end{array}$$

(8)

where N$(a,b)$ denotes the normal distribution with the mean a and the variance b. The details of the calculations are given in Appendix A. If $\tau$ is finite, one has $D_h\left(t\right)\simeq N\tau /2$ for $t\gg \tau \gg 1$. In the limit $\tau \to \infty$, the pheromone does not evaporate and then $D_h\left(t\right)=Nt$.

The essential differences between $Z\left(t\right)$ and $Z(m,t)$ include a shift in the expected value by $1/2N$ and the presence of a factor of N in the numerator of the variance of $Z(m,t)$. The shift of $1/2N$ arises from the finding that $X(m,s)$ in $S(m,t)$ is 1, which is larger than $E\left[X\right(i,s\left)\right]=1/2$ for $i\ne m$. The value of the pheromone contains information about the correct choice, leading to $E\left[S(m,t)\right]>\frac{1}{2}E\left[S\left(t\right)\right]$. However, in the “cheating” process, the variables $X(i,s)$, are combined by $X(m,s)$, as in Equation (2), resulting in a larger variance for $S(m,t)$. The factor of N in the numerator of the variance of $Z(m,t)$ is a consequence of this combination process.

From the distribution of $Z(m,t_0)$, one can determine the values of $\tau$ or $t_0$ that guarantee that $Z(m,t_0)$ is greater than 0.5 for the limits $t\gg \tau \gg 1$ and $\tau \to \infty$, respectively. With a confidence level of 1%, $\tau$ and $t_0$ should satisfy the following conditions:

$$\begin{array}{ccc}& & t\gg \tau \gg 1:\frac{1}{2N}\ge \frac{2.58}{2\sqrt{2\tau }}\to \tau \ge 3.33N^2,\hfill \\ & & \tau \to \infty :\frac{1}{2N}\ge \frac{2.58}{2\sqrt{t_0}}\to t_0\ge 6.66N^2.\hfill \end{array}$$

3.2. Case with $\tau \to \infty$

We take the limit $\tau \to \infty$ in Equations (5) and (7). We replace $D(t+1)$ and $D_h(t+1)$ with $N(t+1)$ and $N(t+1)$, respectively. This results in the following equations:

$$\begin{array}{ccc}\hfill dZ(m,t)& =& \frac{\overline{f\left(Z\right(i,t\left)\right)}}{(t+1)Z(t+1)}\left[\left(1+\frac{1-f\left(Z\right(m,t\left)\right)}{N\overline{f\left(Z\right(i,t\left)\right)}}\right)f\left(Z(m,t)\right)-Z(m,t)\right]dt\hfill \\ & +& \left(\frac{\overline{f\left(Z\right(i,t\left)\right)}}{(t+1)Z(t+1)}\right)\sqrt{f\left(Z\right(m,t\left)\right[1-f\left(Z\right(m,t\left)\right)]}dW\left(t\right),m=1,\cdots ,N,\hfill \\ \hfill \frac{d}{dt}Z\left(t\right)& =& \frac{1}{(t+1)}(\overline{f\left(Z\right(i,t\left)\right)}-Z\left(t\right)).\hfill \end{array}$$

The initial conditions of $Z\left(t\right)$ and $Z(m,t)$ at $t=t_0$ are as follows:

$$\begin{array}{ccc}\hfill Z\left(t_0\right)& \sim & N\left(\frac{1}{2},\frac{1}{4N{t}_0}\right),\hfill \\ \hfill Z(m,t_0)& \sim & N\left(\frac{1}{2}+\frac{1}{2N},\frac{1}{4t_0}\right).\hfill \end{array}$$

In the adiabatic approximation, where the time development of $Z(m,t)$ is much faster than that of $Z\left(t\right)$, the structure of the SDE for each $Z(m,t)$, where $m=1,\cdots ,N$, is the same as that of a nonlinear P’olya urn [33,34]. The probability for $Z(m,t)$ to converge to a stable solution of the following equation is positive [35],

$$\left(1+\frac{1-f\left(z\right)}{N\overline{f\left(Z\right(i,t\left)\right)}}\right)f\left(z\right)=z.$$

(9)

Here, a stable (unstable) solution of Equation (9) means that the curve of the left-hand-side of the equation crosses the diagonal curve $y=z$ in in the downward (upward) direction [35]. We denote the stable and unstable solutions as $z_s$ and $z_u$, respectively.

In the limit $N\to \infty$, the positive driving force $\frac{1-f\left(z\right)}{N\overline{f\left(Z\right(i,t\left)\right)}}$ in Equation (9) disappears. The system becomes $Z_2$-symmetric, and $z=1/2$ is a the solution of $f(1/2)=1/2$. If N is finite, the positive driving force breaks the $Z_2$ symmetry. We plot $(1+b(1-f\left(z\right)\left)f\right(z)$ in Figure 1. $b>0$ corresponds to $1/N\overline{f\left(Z\right(i,t\left)\right)}$ and $b\in [0.01,0.02$] for $N=100$ and $\overline{f\left(Z\right(i,t)}\in [1/2,1]$; $b=0$ and $b=0.2$ are used in Figure 1, left and right, respectively, along with $ϵ=0.1$.

In the $Z_2$-symmetric case ($b=0$), the stability of the solution $z=1/2$ depends on the slope of $f\left(z\right)$ at $z=1/2$. As $f^{\prime }(1/2)=(1-ϵ)\alpha$, where the prime denotes z-derivative, the critical value of $\alpha$ is ${\alpha }_c=1/(1-ϵ)$. If $\alpha <(>){\alpha }_c$, $z=1/2$ is (un)stable. If $\alpha ={\alpha }_c$, the curve of $f\left(z\right)$ is tangential to the diagonal. $Z(m,t)$ converges to $1/2$ for $\alpha \le {\alpha }_c$, as $z=1/2$ is the unique stable solution $z_s$. For $\alpha >{\alpha }_c$, there appears two stable solutions $z_{s1},z_{s2}$: $z_{s1}$ is within $(0,0.5)$, and $z_{s2}$ is within $(0.5,1.0)$. $z=0.5$ becomes the unstable solution $z_u$.

The dotted lines in Figure 2 show the solutions versus $\alpha$ for the $Z_2$-symmetric case. For $\alpha <{\alpha }_c$, $z=1/2$ (black dotted line) is the stable solution. For $\alpha >{\alpha }_c$, $z=1/2$ (gray dotted line) becomes unstable ($z_u=1/2$) and two stable solutions $z_{s1},z_{s2}$ depart from $z=1/2$ continuously with $\alpha >{\alpha }_c$. The stable solution to which $Z(m,t)$ converges depends on the initial value of $Z(m,t_0)$. In general, if $Z(m,t_0)$ is greater (smaller) than $1/2$, the probability of the convergence to 1 is greater (smaller) than $1/2$. $z_u$ determines the “attractive domains” for the stable solutions $z_{s1},z_{s2}$. The susceptibility of the expected value of $Z(m,t)$ to the initial value $Z(m,t_0)$ is the order parameter of the nonlinear Pólya urn [33,34]. As the order parameter is proportional to the difference in the two stable states $z_{s1},z_{s2}$, the order parameter is a continuous function of $\alpha$, and the phase transition is continuous.

In the $Z_2$-asymmetric case ($b\ne 0$), for $\alpha <{\alpha }_c$, there is a stable solution, $z_s$, in the z-range $(0.5,1)$. $z_s$ increases with $\alpha$; at some critical value, ${\alpha }_c$, of $\alpha$, the curve $f\left(z\right)(1+b(1-f\left(z\right))$ becomes tangential to the diagonal at some $z_t$. $z_t$ is known as a touchpoint and to be stable [36]. As $\alpha$ continues to increase beyond ${\alpha }_c$, two significant changes occur: a new stable solution, $z_{s1}$, emerges from the touchpoint $z_t$, while an unstable solution, $z_u$, also becomes apparent.

When $\alpha <{\alpha }_c$, only one stable solution, $z_s$, exists within the range $1/2<z_s<1$. Conversely, for $\alpha >{\alpha }_c$, the specific stable solution to which $Z(m,t)$ converges depends on the initial values of $Z(m,t_0)$. At $\alpha ={\alpha }_c$, both the stable fixed point $z_s$ and the touchpoint $z_t$ remain stable. The solution to which $Z(m,t)$ converges is determined by the initial values of $Z(m,t_0)$ in this case as well. For $\alpha >{\alpha }_c$, the situation mirrors that of $\alpha ={\alpha }_c$. Once $\alpha \ge {\alpha }_c$, the order parameter becomes positive, and the phase transition becomes discontinuous.

Figure 3 shows the results of the numerical studies in the limit $\tau \to \infty$. We sampled a trajectory of $Z(m,t)$ and $Z\left(t\right)$ for $1\le t\le {10}^9$ with $N={10}^2$ and $ϵ=0.01$. Figure 3 presents the distribution of $Z(m,t_0)$ for two different values of $t_0$, namely, $t_0\in {10}^3,{10}^6$. The mean value of $Z(m,t_0)$ is approximately $1/2+1/2N$, which aligns with the theoretical predictions. The variance of $Z(m,t_0)$ is given by $1/4t_0$, so the variance for $t_0={10}^3$ is about ${10}^3$ times larger than that for $t_0={10}^6$. Consequently, if $t_0={10}^3$ is chosen, a significant proportion of $m\in 1,\cdots ,N$ has $Z(m,t_0)<z_u\simeq 0.5$, resulting in a high probability that $Z(m,t)$ converges to $z_s^{\prime }<1/2$ for $\alpha =2.0$. In such cases, $Z\left(t\right)$ cannot reach 1 due to the convergence of $Z(m,t)$ to $z_s^{\prime }$. On the other hand, if $t_0={10}^5$ is set, the ratio of $m\in 1,\cdots ,N$ with $Z(m,t_0)<z_u\simeq 0.5$ is zero, ensuring that $Z(m,t)$ always converges to $z_s>1/2$. As a result, $Z\left(t\right)$ monotonically increases toward 1 for $\alpha =2.0$. For $\alpha =1.0<{\alpha }_c$, where only one stable state, $z_s\simeq 1$, exists, $Z(m,t)$ consistently converges to $z_s$.

One can see that $Z\left(t\right)$ monotonically approaches $z_s$ with time for both $t_0={10}^3$ and $t_0={10}^5$ cases within the range $t\le {10}^9$. In the case of $\alpha =0.5$, where $z_s\simeq 0.5$, $Z\left(t\right)$ experiences relatively little change.

3.3. Case with $\tau <\infty$

In the case where $\tau$ is finite, let us make the assumption that $t\gg \tau \gg 1$ and replace $D(t+1)$ with $N\tau$ in Equations (5) and (7). This leads to the following equations:

$$\begin{array}{ccc}\hfill dZ(m,t)& =& \frac{\overline{f\left(Z\right(i,t\left)\right)}}{\tau Z(t+1)}\left[\left(1+\frac{1-f\left(Z\right(m,t\left)\right)}{N\overline{f\left(Z\right(i,t\left)\right)}}\right)f\left(Z(m,t)\right)-Z(m,t)\right]dt\hfill \\ & +& \frac{\overline{f\left(Z\right(i,t\left)\right)}}{\tau Z(t+1)}\sqrt{f\left(Z\right(m,t\left)\right)(1-f(Z(m,t))}dW\left(t\right),m=1,\dots ,N,\hfill \\ \hfill \frac{d}{dt}Z\left(t\right)& =& \frac{1}{\tau }(\overline{f\left(Z\right(i,t\left)\right)}-Z\left(t\right)).\hfill \end{array}$$

(10)

The initial conditions for $Z\left(t_0\right)$ and $Z(m,t_0)$ at $t_0\gg \tau$ are as follows:

$$\begin{array}{ccc}\hfill Z\left(t_0\right)& \sim & N\left(\frac{1}{2},\frac{1}{8N\tau }\right),\hfill \\ \hfill Z(m,t_0)& \sim & N\left(\frac{1}{2}+\frac{1}{2N},\frac{1}{8\tau }\right).\hfill \end{array}$$

(11)

The dynamics of $Z(m,t),m=1,\dots ,N$, are coupled through $Z\left(t\right)$ and $\overline{f}\left(Z(i,t)\right)$. To simplify and analyze this coupled system, we focus on the stationary state of $Z(m,t)$ in the limit $t\to \infty$.

$Z(m,t)$ are expected to fluctuate around the stable fixed points of $f\left(z\right)$ in the stationary state. As was observed in Section 3.2 for the case of $\tau \to \infty$, for $\alpha <{\alpha }_c$, there is only one stable fixed point, and for $\alpha \ge {\alpha }_c$, two stable fixed points exist, one of which is near one. The stationary distribution is unimodal for $\alpha <{\alpha }_c$ and bimodal for $\alpha \ge {\alpha }_c$. Let us denote the stationary distribution and the mean value of $Z(m,t)$ as $P_{\mathrm{st}}\left(z\right)$ and ${\mu }_{\mathrm{st}}$, respectively. As $f\left(Z\right(m,t\left)\right)$ is the probability for $X(m,t+1)=1$, one can assume $\overline{f\left(Z\right(i,t)}=Z\left(t\right)={\mu }_{\mathrm{st}}$ in the stationary state. The SDEs in Equation (10) can be simplified as follows when replacing $\overline{f}\left(Z(i,t)\right)$ and $Z\left(t\right)$ with ${\mu }_{\mathrm{st}}$:

$$\begin{array}{ccc}\hfill dZ(m,t)& =& \frac{1}{\tau }\left[\left(1+\frac{1-f\left(Z\right(m,t\left)\right)}{N{\mu }_{\mathrm{st}}}\right)f\left(Z(m,1)\right)-Z(m,t)\right]dt\hfill \\ & +& \frac{1}{\tau }\sqrt{f\left(Z\right(m,t\left)\right)(1-f(Z(m,t))}dW\left(t\right)\hfill \\ & =& A\left(Z\right(m,t\left)\right)dt+B\left(Z\right(m,t\left)\right)dW\left(t\right),\hfill \\ \hfill A\left(z\right)& =& \frac{1}{\tau }\left[\left(1+\frac{1-f\left(z\right)}{N{\mu }_{\mathrm{st}}}\right)f\left(z\right)-z\right],\hfill \\ \hfill B\left(z\right)& =& \frac{1}{\tau }\sqrt{f\left(z\right)(1-f(z)}.\hfill \end{array}$$

(12)

The stationary state with reflecting boundary conditions is determined by a potential solution [37], which can be expressed as:

$$\begin{array}{ccc}\hfill P_{\mathrm{st}}\left(z\right)& \propto & \frac{1}{B^2\left(z\right)}exp\left({\int }_{1/2}^z\frac{2A\left(y\right)}{B^2\left(y\right)}dy\right),\hfill \\ \hfill \frac{2A\left(y\right)}{B^2\left(y\right)}& =& 2\tau \frac{f\left(y\right)-y}{f\left(y\right)(1-f(y\left)\right)}+\frac{2\tau }{N{\mu }_{\mathrm{st}}}.\hfill \end{array}$$

(13)

The second term, $2\tau /N{\mu }_{\mathrm{st}}$, arises from the $Z_2$ symmetry-breaking field and causes a shift in the stationary distribution in the positive direction.

Figure 4 shows $P_{\mathrm{st}}\left(z\right)$ (13) for $ϵ=0.01$ and $N={10}^2$; ${\mu }_{\mathrm{st}}$ is chosen so that the mean value of $P_{\mathrm{st}}\left(z\right)$ coincides with

$${\mu }_{\mathrm{st}}={\int }_0^1{P}_{\mathrm{st}}\left(z\right)zdz.$$

The pair of the parameters $(\alpha ,{\mu }_{\mathrm{st}})$ are $(0.0,0.50)$, $(0.5,0.51)$, $(0.9,0.54)$, $(0.99,0.67)$, $(1/0.99,0.74)$, and $(2.0,0.54)$. As $\alpha$ increases, the peak position shifts in the positive z-direction, which can be expected by the dependence of the stable solution $z_s$ on $\alpha$ in Figure 2. If $\alpha =1/(1-ϵ)$, the peak appears at $z=1$, since there is only one stable fixed point near $z=1$ in Figure 1. When $\alpha =2$, there are two stable fixed point, and the stationary distribution is bimodal.

In order to derive the dependence of ${\mu }_{\mathrm{st}}$ and the variance of $P_{\mathrm{st}}\left(z\right)$ on $\alpha$, let us assume that $Z(m,t)$ fluctuates around ${\mu }_{\mathrm{st}}\simeq \frac{1}{2}$ for $\alpha <{\alpha }_c$. We linearize $f\left(z\right)$ in the vicinity of $z=1/2$ as

$$f\left(z\right)=\frac{1}{2}+(1-ϵ)\alpha \left(z-\frac{1}{2}\right).$$

Let us also approximate $B^2\left(z\right)$ as ${\mu }_{\mathrm{st}}(1-{\mu }_{\mathrm{st}})=1/4$, then $P_{\mathrm{st}}\left(z\right)$ becomes

$$P_{\mathrm{st}}\left(z\right)\propto exp\left\{-\frac{{\left[z-\left(\frac{1}{2}+\frac{1}{2N(1-(1-ϵ\left)\alpha \right)}\right)\right]}^2}{\frac{2}{8\tau (1-(1-ϵ\left)\alpha \right)}}\right\}.$$

In the case with $\alpha =0$, the ants do not observe the information of the pheromones and decide by themselves. The expected value and the variance are consistent with the results (11) for the initial state. The expected value and the variance increase with $\alpha$ for $0\le \alpha <1/(1-ϵ)$.

The shape of the stationary distribution changes from the monomodal shape for $\alpha <{\alpha }_c$ to the bimodal shape for $\alpha \ge {\alpha }_c$. Figure 5 shows the stationary distribution $P_{\mathrm{st}}\left(z\right)$ of $Z(m,t)=z$ for $\alpha \in \{0,0.5,0.9,0.99,1/0.99,2.0\},ϵ=0.01,\tau =100$ and $N={10}^2$. We also plot $P_{\mathrm{st}}\left(z\right)$ in Equation (13) with solid-line curves. Except the $\alpha =2.0$ case, the numerical results agree with the theoretical ones. As $\alpha$ increases from 0 to $1/0.99$, the mean value and the variance of $Z(m,t)$ increases. For $\alpha =1/0.99$, the distribution of $Z(m,t)$ has a peak at $z=1$. The distribution becomes bimodal and has two peaks near $z=0$ and $z=1$ for $\alpha =2$. For $\alpha =2.0$, $P_{\mathrm{st}}\left(z\right)$ becomes bimodal and the equilibration time to reach the stationary state becomes quite long. We beleive the latter is the reason for the discrepancy between the numerical and theoretical results.

4. Discussion

In the presented paper, we studied a simple model for ACO and the convergence properties of the solutions. Ants answer many two-choice questions in sequence and deposit pheromone as they choose. As the amount of the pheromones is the number of correct answers, the following ants can receive information or hints about the correct choices. We showed that the model reduces to a multivariate nonlinear Pólya urn process and the pheromones break the $Z_2$ symmetry of the process. By varying the exponent $\alpha$ of the decision function of the ants, there occurs a phase transition about the convergence of the probability of choosing the correct answer for each question in the limit $\tau \to \infty$. For $\tau <\infty$, the change in the stationary distribution between the monomodal and the bimodal shape occurs as we vary $\alpha$.

Previous studies adopted values of $\alpha =1$ or smaller in solving actual optimization problems, like the TSP [24]. In $\alpha$-annealing, $\alpha$ increases gradually, as shown in Ref. [24]. In our study, we showed that the duration of the period $\alpha =0$ should be long enough to ensure that the initial value of $Z(m,t)$ is in the attractive domain of the suitable stable state ($Z(m,t)>z_u$) in the case when $\tau =\infty$. Subsequently, with $\alpha \ge 1$ in effect, $Z\left(t\right)$ converges to a value close to 1. In the case of $\tau <\infty$, the timescale for pheromone evaporation, represented by $\tau$, shown to be sufficiently long to maintain the same initial conditions. However, $Z(m,t)$ does not converge to a specific value; instead, it follows a stationary distribution that exhibits both bimodal and monomodal shapes depending on the value of $\alpha$. To achieve a distribution of $Z\left(t\right)$ with a prominent peak near $z=1$, the $\alpha$-annealing process is an effective strategy. Both the stable solution $z_s$ and the distribution of $Z(m,t)$ suggest that after a lengthy enough period with $\alpha =0$, it is advantageous to gradually increase $\alpha$ from 1. However, the efficiency of the annealing process depends on the specific problem being addressed. Future studies are considered to clarify on an efficient $\alpha$-annealing schedule.