A Phase Transition Control Framework for UAV Swarms Inspired by Pigeon Roosting Behavior

Highlights

What are the main findings?

• A bio-inspired phase transition control framework with two distinct motion phases is proposed for UAV swarms, combining a self-propulsion term, an interaction potential term, and a roosting force term.
• The existence of the translational and vortex motion phases are proven, and the stability properties of the two motion phases are discussed based on Lyapunov theory.

What are the implications of the main findings?

• The proposed framework provides a phase transition mechanism triggered by the roosting force term, enabling UAV swatms to switch between different motion phases.
• Numerical simulations validate the stability of the two motion phases, and display the transition process triggered by the roosting force term, demonstrating the framework’s potential for enhancing swarm adaptability in complex missions.

Abstract

This study proposes a bio-inspired control framework for unmanned aerial vehicle (UAV) swarms, designed to emulate the collective motion phase transitions observed in the homing behavior of pigeon flocks. A second-order self-propelled particle model is established, integrating a self-propulsion term, an interaction potential term, and a key roosting force term inspired by the roosting behavior of pigeons. The framework enables the swarm to dynamically switch between a translational motion phase and a vortex motion phase based on the distance to a designated roost location. Based on the proposed swarm model, theoretical analysis proves the stability property of the specific two motion phases under specific conditions. Numerical simulations validate the stability of the two motion phases, demonstrating that UAV swarms can reliably maintain each phase and execute phase transitions triggered by the roosting force. The proposed framework is able to describe the phase transition behavior in the process of pigeons returning home.

1. Introduction

Multi-agent systems have attracted great interest from many researchers, with related findings widely applied to the control of multi-robot systems [1,2,3,4] and unmanned aerial vehicle (UAV) swarms [5,6,7,8,9]. In nature, many animals exhibit intelligent and robust coordinated collective behaviors through years of evolution [10], such as bird flocks [11,12], insect swarms, fish schools [13] and wolf packs [14]. The study of these biological animals serves as a rich source of inspiration for multi-agent systems, providing insights that enhance decision-making, formation control and communication mechanisms. Investigating multi-agent systems through the lens of biological collective behavior thus presents distinct advantages for developing robust coordination strategies.

Birds are commonly observed in large groups, and various models have been proposed to explain their specific behaviors, such as pigeons [15], starlings [16], and jackdaws [17,18]. Their flocking behaviors provide important biological inspiration in multi-agent systems, especially for UAVs [19,20]. Among these, one of the most essential models is the self-propelled particle (SPP) model, which holds significant foundational and application value for understanding and designing swarm systems.

There are various models describing self-propelled particles [21,22]. The main feature of SPP is that each individual can maintain its own velocity while interacting with its neighbors to achieve group consensus. One of the most widespread models is the Vicsek model, which contains a velocity alignment term and a disturbance term [23]. Due to its simplicity and capacity to exhibit transitions between different motion phases, it has been extensively studied [24,25,26]. Another main category of SPP models is the Boid model (or Reynolds model) [27], which employs a second-order dynamic model to simulate physical motion. It introduces a self-propulsion term into individuals’ acceleration to maintain individual speed while utilizing three fundamental rules (attraction, repulsion, and alignment) to achieve collective consensus [28,29,30]. Compared with the Vicsek model, the Boid model is more frequently studied in real multi-robot systems, as the second-order dynamic model captures the kinematic characteristics of robot platforms [31,32].

Phase transition control in robot swarms has attracted growing interest, as it can significantly enhance the swarm’s adaptability in complex environments and different mission requirements. Although a universally accepted definition is still under discussion, phase transition control in multi-agent systems generally refers to the switch between distinct motion modalities driven by internal dynamics or external perturbations. Introducing phase transition control into UAV swarms offers a promising approach to enhancing adaptability in increasingly complex operational environments, thereby improving mission robustness. To achieve this in UAV swarms, research must be grounded in models that inherently possess the potential for rich phase behaviors and transitions.

Self-propelled particle models exhibit rich phase transition phenomena and are capable of generating diverse motion patterns. The Vicsek model was among the first to demonstrate phase transition behavior, revealing that complex collective motion can emerge from simple local interaction rules [23,33]. Compared to the Vicsek model, the Boid model incorporates richer interaction mechanisms and thus has greater potential to exhibit diverse motion phases, making it widely used in studies of multi-agent phase transitions.

Subsequent research has focused on characterizing these phase transitions and developing analytical tools to understand their underlying mechanisms. Hindes [22] studied the motion of self-propelled particle systems in three-dimensional space and employed numerical methods to compute critical parameters for transitions from aggregation to dispersion. Cheng et al. [34] investigated motion phases under different potential functions and interaction ranges, analyzing how potential parameters influence collective behavior. Another study [35] explored motion phases under communication delays, where mean-field approximation and bifurcation theory were used to identify two stable motion phases. Popovici [36] proved that the translating state is asymptotically stable in self-propelled models with consensus terms. Orsogna et al. [37] analyzed various motion phases under soft-core interactions, particularly using the Morse potential framework.

Beyond describing the motion phases formed from initial random states under different parameters, related literature further discusses how certain motion phases disintegrate and transition into others. Zhang et al. [38] conducted extensive simulations to examine the transition from vortex to crystal motion states in a collective escaping from an external predator. They analyzed the effects of velocity alignment and the attraction–repulsion ratio on the critical danger radius. Hindes et al. [39] studied the transition from translational to vortex phases in two groups with identical control laws, exploring how the distance between the centers of the two groups influences the final stable collective state.

More importantly, the mechanisms for actively controlling and inducing transitions between these phases within a unified dynamical framework remain largely unexplored.

Despite the widely observed fact that self-propelled particle models can exhibit diverse motion phases under different parameters, there is a lack of theoretical understanding of the existence and stability of these motion phases for models with potential functions. More importantly, the mechanisms for actively controlling and switching between these phases within a unified dynamical framework are comparatively unexplored.

To this end, this paper introduces a specific second-order SPP model augmented with a bio-inspired roosting force, designed explicitly to emulate the homing behavior of pigeon flocks. Furthermore, instead of analyzing individual agent dynamics in isolation, we adopt a centroid-based perspective that enables a unified characterization of collective motion at the group level. This approach allows us to systematically derive conditions under which the swarm exhibits translational or vortex-like behavior and to analyze their stability within a common theoretical framework.

The main contribution of the paper is listed as follows:

• A bio-inspired roosting force term is introduced as a continuous control input to emulate homing behavior. Numerical simulations show that by modulating a single scalar parameter, the swarm can achieve stable and smooth transitions between translational and vortex motion phases.
• A Lyapunov-based stability analysis for both the translational and vortex motion phases is performed. For the complex vortex phase, we establish and prove a non-trivial sufficient stability condition involving the commutativity of the potential’s Hessian with the rotation matrix. This criterion offers concrete theoretical guidance for potential function design in practical engineering applications.
• The generality of the proposed framework is further validated by extending the interaction model beyond the quadratic potential. In particular, simulations using a Morse-type potential demonstrate that vortex-like motion and phase transitions can still emerge even when the conditions analyzed by the Lyapunov method is not strictly satisfied.

The remainder of the paper is organized as follows. Section 2 proposes the swarm model and provides the basic assumption of the potential function, while Section 3 discusses the derivation of the two phases of the swarm model. The stability of the two phases is proved in Section 4, and Section 5 uses simulation to prove the results. Finally, Section 6 contains our conclusion.

2. Materials and Methods

This study presents a framework inspired by the homing behavior of pigeon flocks, drawing fundamental principles from self-propelled particle systems observed in collective animal behavior [27]. Empirical studies reveal that during homing, a flock of pigeons may exhibit two distinct collective motion phases illustrated in Figure 1: a long-range translational phase in which the flock moves cohesively toward the loft [40,41], and a near-home vortex phase in which the flock circulates above the loft before landing [15,42]. This phase transition is governed by an intrinsic roosting instinct, that generates a steering force guiding individuals toward the destination.

To emulate this biological mechanism, we develop a second-order self-propelled particle (SPP) model for a UAV swarm. The model integrates three core components: (1) a self-propulsion term that enables individuals to maintain a preferred cruise speed; (2) a pairwise interaction term that ensures flock cohesion and collision avoidance; and (3) a roosting force term that mimics the homing instinct. Crucially, this roosting force acts as the external control input that triggers dynamic transitions between the translational and vortex phases, enhancing the adaptability for complex operational scenarios.

We consider a swarm of N UAVs in a two-dimensional plane, with each UAV fixed at the same height. Additionally, we adopt a fully connected graph topology for the swarm, meaning each agent can communicate with all other agents. This topology ensures global information exchange and facilitates coherent collective behavior. Therefore, the model of the i-th UAV can be written as below:

$${\dot{\mathbf{x}}}_i={\mathbf{v}}_i$$

(1)

$${\dot{\mathbf{v}}}_i=a\left(v_0-\alpha {\left|{\dot{\mathbf{x}}}_i\right|}^2\right){\dot{\mathbf{x}}}_i-{\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)+\lambda {\mathbf{F}}_i^{roost}({\mathbf{x}}_i,{\mathbf{v}}_i)$$

(2)

for $i=1,2,\cdots ,N$. Here, ${\mathbf{x}}_i$ represents the position of the i-th UAV, $a,b>0$ indicate gain coefficients for the self-propulsion term and potential interaction term respectively. The term $a\left(v_0-\alpha {\left|{\dot{\mathbf{x}}}_i\right|}^2\right)$ acts as a speed regulator, which ensures that each UAV’s speed converges to the desired cruise speed, where $v_0>0$ and $\alpha >0$ are the baseline speed scalar and the damping coefficient respectively. The second term $-b{\displaystyle \sum _{j=1,j\ne i}^N}\nabla U\left({\mathbf{x}}_{ij}\right)$ captures the attraction, repulsion, and alignment interaction between UAVs, where $\nabla U\left({\mathbf{x}}_{ij}\right)$ is the gradient of the potential function between agents i and j, where ${\mathbf{x}}_{ij}={\mathbf{x}}_i-{\mathbf{x}}_j$ is the relative position of UAV i and j. The roosting force term $\lambda {\mathbf{F}}_i^{roost}({\mathbf{x}}_i,{\mathbf{v}}_i)$ functions as the key bio-inspired term that emulates the homing instinct. ${\mathbf{F}}_i^{roost}({\mathbf{x}}_i,{\mathbf{v}}_i)$ acts as a steering force that biases the UAV’s motion toward a designated goal, $\lambda \ge 0$ is the intensity of this force, $\lambda =0$ means that there does not have a roosting force term. Functionally, this roosting force serves as the phase transition control input. By activating $\lambda$, the swarm can be driven to switch between the distinct motion phases.

In Section 4, the potential interaction function U is not restricted to a specific form but must satisfies the following properties to ensure physical consistency and stability:

• Isotropy. The potential is a function solely of the inter-agent distance, that is $U\left({\mathbf{x}}_{ij}\right)=f\left(r\right)$, where $r=\left|{\mathbf{x}}_{ij}\right|$. Its gradient is therefore $\nabla U\left({\mathbf{x}}_{ij}\right)=f^{\prime }\left(\left|{\mathbf{x}}_{ij}\right|\right){\displaystyle \frac{{\mathbf{x}}_{ij}}{\left|{\mathbf{x}}_{ij}\right|}}$, which is directed along the line connecting the two agents. This indicates that the interaction force magnitude depends only on the distance between agents, not on their relative orientation.
• Reciprocity. The potential function has a unique minimum at a desired equilibrium distance the potential gradient satisfies $\nabla U\left({\mathbf{x}}_{ij}\right)=-\nabla U\left({\mathbf{x}}_{ji}\right)$, meaning that the interaction is reciprocal between any pair of agents. This ensures the conservation of momentum within the swarm;
• Positive definite gradient. The Hessian matrix of the interaction potential is positive definite for all non-zero relative positions, that is

  $${\mathbf{y}}^T\left[{{\nabla }^2}_{\mathbf{r}}U\left({\mathbf{x}}_{ij}\right)\right]\mathbf{y}>0\forall \mathbf{y}\in {\mathbb{R}}^2\setminus \left\{\mathbf{0}\right\},\forall {\mathbf{x}}_{ij}\ne \mathbf{0}$$

  (3)
  This property ensures that the potential $f\left(\left|{\mathbf{x}}_{ij}\right|\right)$ has a unique global minimum at the desired equilibrium distance $d_s>0$, where $f^{\prime }\left(d_s\right)=0$. This promotes stable spacing between agents, analogous to the tendency of birds in a flock to maintain a specific separation.

The potential function can take various forms. One of the simplest form of the potential function is the quadratic potential function: $U\left({\mathbf{x}}_{ij}\right)={\displaystyle \frac{1}{2}}{\left(\left|{\mathbf{x}}_{ij}\right|-d_s\right)}^2$, and $\nabla U\left({\mathbf{x}}_{ij}\right)={\displaystyle \frac{\left(\left|{\mathbf{x}}_{ij}\right|-d_s\right){\mathbf{x}}_{ij}}{\left|{\mathbf{x}}_{ij}\right|}}$.

Through normalization procedures, the model can be simplified to

$${\dot{\mathbf{x}}}_i={\mathbf{v}}_i$$

(4)

$${\dot{\mathbf{v}}}_i=a\left(1-{\left|{\dot{\mathbf{x}}}_i\right|}^2\right){\dot{\mathbf{x}}}_i-{\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)+\lambda {\mathbf{F}}_i^{roost}({\mathbf{x}}_i,{\mathbf{v}}_i)$$

(5)

This model provides a foundational framework for analyzing phase transitions in swarms, such as shifts from translational motion to vortex motion, through minimal parameter adjustments. The combination of self-propulsion, potential-based interactions, and roosting force enables robust and adaptable swarm behaviors inspired by natural systems. The following analysis will be based on the normalized model (4) and (5).

The core innovation lies in the integration of a phase transition control term alongside conventional flocking mechanisms. This allows the swarm to dynamically alter its collective morphology in response to task requirements, mimicking the behavioral plasticity seen in natural swarms. The schematic of the control framework is shown in Figure 2.

Figure 2 illustrates the overall architecture of the proposed distributed swarm control framework. In this scheme, each agent (UAV) obtains the positions of its neighboring agents through communication topology. Based on this information, it computes several key components: interaction forces for cohesion and collision avoidance, a velocity-dependent self-propulsion term, and an additional roosting force that attracts the agent toward a designated roost location (here simplified as the origin). The emergent collective behavior under this control model leads to two distinct motion phases, which are analyzed in detail in the subsequent sections.

3. Analysis of Motion Phases

Two motion phases can be investigated using the model (4) and (5). Inspired by reference [35], we analyze the collective dynamics by adopting a coordinate frame centered at the swarm’s center of mass when $\lambda =0$. This approach effectively decouples the motion of swarm center from the relative motion of agents, facilitating a clearer examination of swarm motion phases.

The position of the swarm’s center is defined as

$$\mathbf{R}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathbf{x}}_i$$

(6)

$${\mathbf{x}}_i=\mathbf{R}+\delta {\mathbf{r}}_i$$

(7)

where $\delta {\mathbf{r}}_i$ represents the displacement of the i-th agent from the center of mass.

By substituting the above into the swarm model (4) and (5) and summing over all individuals, we obtain

$$N\ddot{\mathbf{R}}=\sum _{j=1}^{N}a\left[1-{\left(\dot{\mathbf{R}}+\delta {\dot{\mathbf{r}}}_j\right)}^2\right]\left(\dot{\mathbf{R}}+\delta {\dot{\mathbf{r}}}_j\right)-\sum _{j=1}^N{\displaystyle \frac{b}{N}}\sum _{k\ne j}\nabla U\left({\mathbf{x}}_{jk}\right)+\sum _{j=1}^N\lambda {\mathbf{F}}_j^{roost}$$

(8)

As the interaction forces are pair-wise, when the following condition $\nabla U\left({\mathbf{x}}_{ij}\right)=-\nabla U\left({\mathbf{x}}_{ji}\right)$ holds, we have

$$\sum _{j=1}^N\sum _{k=1,k\ne j}^N\nabla U\left({\mathbf{x}}_{jk}\right)=0$$

(9)

After simplification, the final equations are

$$\ddot{\mathbf{R}}=a\dot{\mathbf{R}}\left(1-{\dot{\mathbf{R}}}^2-{\displaystyle \frac{1}{N}}\sum _{j=1}^N\delta {{\dot{\mathbf{r}}}_j}^2\right)+{\displaystyle \frac{a}{N}}\sum _{j=1}^N\left[1-\left(2\dot{\mathbf{R}}\delta {\dot{\mathbf{r}}}_j+\delta {{\dot{\mathbf{r}}}_j}^2\right)\right]\delta {\dot{\mathbf{r}}}_j+{\displaystyle \frac{1}{N}}\sum _{j=1}^N\lambda {\mathbf{F}}_j^{roost}$$

(10)

$$\begin{gathered} \delta {\ddot{\mathbf{r}}}_i=a\left(-2\dot{\mathbf{R}}\delta {\dot{\mathbf{r}}}_i-\delta {{\dot{\mathbf{r}}}_i}^2+{\displaystyle \frac{1}{N}}\sum _{j=1}^N\delta {{\dot{\mathbf{r}}}_j}^2\right)\dot{\mathbf{R}}+a\left(1-{\left(\dot{\mathbf{R}}+\delta {\dot{\mathbf{r}}}_i\right)}^2\right)\delta {\dot{\mathbf{r}}}_i \\ -{\displaystyle \frac{a}{N}}\sum _{j=1}^N\left[1-\left(2\dot{\mathbf{R}}\delta {\dot{\mathbf{r}}}_j+\delta {{\dot{\mathbf{r}}}_j}^2\right)\right]\delta {\dot{\mathbf{r}}}_j+\lambda {\mathbf{F}}_i-{\displaystyle \frac{1}{N}}\sum _{j=1}^N\lambda {\mathbf{F}}_j^{roost} \end{gathered}$$

(11)

Employing the mean-field approximation $N\to \infty$, neglecting the fluctuation term $\delta {\mathbf{r}}_i$ [35] and setting the roosting force coefficient $\lambda =0$, a simplified equation near the equilibrium point can be derived:

$$\ddot{\mathbf{R}}=a\dot{\mathbf{R}}\left(1-{\dot{\mathbf{R}}}^2\right)$$

(12)

Thus, two equilibrium solutions $\dot{\mathbf{R}}=\mathbf{0}$ and $\left|\dot{\mathbf{R}}\right|=1$ are obtained, representing the vortex motion phase and the translational motion phase respectively.

1. Vortex Motion Phase

Substituting $\dot{\mathbf{R}}=\mathbf{0}$ and $\lambda =0$ into (11) yields the relative dynamics of each agent for this phase:

$$\delta {\ddot{\mathbf{r}}}_i=a\left(1-\delta {{\dot{\mathbf{r}}}_i}^2\right)\delta {\dot{\mathbf{r}}}_i-{\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)$$

(13)

In this phase, individual agents exhibit stable circular motion around the stationary center of mass. The self-propulsion term regulates individual speed, while the potential forces maintain cohesive spacing, leading to a vortical state.

Remark 1. 

Another possible solution that satisfies $\dot{\mathbf{R}}=\mathbf{0}$ is the trivial equilibrium ${\dot{\mathbf{x}}}_i=0$ for all agents. However, due to its instability, it is not considered in this work.

2. Translational Motion Phase

This state is characterized by a swarm center moving with constant velocity $\mathbf{R}={\mathbf{V}}_{R}t+{\mathbf{R}}_0$, where the center of mass moves with a constant velocity $\left|{\mathbf{V}}_R\right|=1$. The relative motion of each agent is governed by

$$\delta {\ddot{\mathbf{r}}}_i=-a\left(2\dot{\mathbf{R}}\delta {\dot{\mathbf{r}}}_i+\delta {{\dot{\mathbf{r}}}_i}^2\right)\left(\dot{\mathbf{R}}+\delta {\dot{\mathbf{r}}}_i\right)-{\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)$$

(14)

This equation admits a stable solution $\delta {\dot{\mathbf{r}}}_i=0$, with the interaction forces balancing out, i.e., ${\displaystyle \frac{b}{N}}{\displaystyle \sum _{j=1,j\ne i}^N}\nabla U_{ij}\left({\mathbf{x}}_{ij}\right)=0$.

In this motion phase, all agents move in a coherent, parallel formation without relative motion, corresponding to a translational state of the swarm.

Therefore, in the action of the phase transition control law (5) and the roosting force deactivated, the swarm exhibits two distinct phase: vortex phase and translational phase.

4. Main Results

In this section, the stability of the two motion phases under the action of control law (5) is discussed. To better illustrate the main results, the following Lemma is given:

Lemma 1. 

Given that $\nabla U\left({\mathbf{x}}_{ij}\right)=-\nabla U\left({\mathbf{x}}_{ji}\right)$ holds and within a fully connected topology, it holds that

$$\sum _{i=1}^N\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)·{\dot{\mathbf{x}}}_j=-\sum _{i=1}^N\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)·{\dot{\mathbf{x}}}_i$$

(15)

Proof. 

$$\begin{array}{cc}\hfill \sum _{i=1}^N\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)·{\mathbf{v}}_j& =\sum _{j=1}^N\sum _{i=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)·{\mathbf{v}}_j\hfill \\ & =\sum _{i=1}^N\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ji}\right)·{\mathbf{v}}_i\hfill \\ & =-\sum _{i=1}^N\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)·{\mathbf{v}}_i\hfill \end{array}$$

(16)

□

The core feature of Lemma 1 is that collective work rate of interaction forces on agent velocities can be equivalently expressed with an opposite sign, employing the antisymmetry property of the potential gradient. It establishes a fundamental property for the summation of pairwise interaction within a fully connected swarm, which will be used in the stability analysis of the vortex motion phase.

Theorem 1. 

Under the action of the phase transition control law (5), the swarm is asymptotically stable near the translational phase solution defined by (14), when the control parameter $\lambda =0$.

Proof. 

In Section 3, we have shown that within the translational phase, each agent moves with a common velocity ${\mathbf{V}}_R$, satisfying $\left|{\mathbf{V}}_R\right|=1$. To analyze deviations from this equilibrium, error variables are defined relative to the center of mass. The position error for the i-th agent is expressed as

$${\mathbf{e}}_i=\delta {\mathbf{r}}_i-\delta {\mathbf{r}}_i^{*}={\mathbf{x}}_i-{\mathbf{x}}_i^{*}$$

(17)

where ${\mathbf{x}}_i^{*}={\mathbf{x}}_i^{*}\left(t\right)$ denotes the time-varying desired position of the agent in the global coordinate system, and $\delta {\mathbf{r}}_i^{*}$ represents the desired position relative to the center of mass. As the entire swarm translates uniformly in the translational phase, the time derivative of the desired relative coordinate in the center-fixed frame is zero, $\delta {\dot{\mathbf{r}}}_i^{*}={\dot{\mathbf{x}}}_i-\dot{\mathbf{R}}={\mathbf{V}}_R-{\mathbf{V}}_R=0$. Consequently, the velocity error in the center-fixed frame simplifies to ${\dot{\mathbf{e}}}_i=\delta {\dot{\mathbf{r}}}_i-\delta {\dot{\mathbf{r}}}_i^{*}=\delta {\dot{\mathbf{r}}}_i$.

Similarly, letting ${\mathbf{v}}^{*}={\mathbf{V}}_R$ express the velocity error in the global coordinate system yields ${\mathbf{v}}_i-{\mathbf{v}}^{*}=\dot{\mathbf{R}}+\delta {\dot{\mathbf{r}}}_i-\left(\dot{\mathbf{R}}+\delta {\dot{\mathbf{r}}}_i^{*}\right)={\dot{\mathbf{e}}}_i$. The acceleration error is then derived as: ${\ddot{\mathbf{e}}}_i={\dot{\mathbf{v}}}_i-{\dot{\mathbf{v}}}^{*}={\dot{\mathbf{v}}}_i={\ddot{\mathbf{x}}}_i$, as ${\dot{\mathbf{v}}}^{*}=0$ due to the constant translational motion of the swarm. The above formulation ensures consistency between the global and center-fixed coordinate systems while accounting for the time-varying nature of the desired positions.

To analyze local stability, we linearize the nonlinear system dynamics around the equilibrium point: ${\mathbf{e}}_i=0,{\dot{\mathbf{e}}}_i=0$. For the nonlinear self-propulsion term, substituting ${\dot{\mathbf{x}}}_i={\mathbf{v}}^{*}+{\dot{\mathbf{e}}}_i$ and expanding the self-propulsion term yields

$$\begin{array}{c}a\left(1-{\left|{\dot{\mathbf{x}}}_i\right|}^2\right){\dot{\mathbf{x}}}_i=a\left(1-{\left|{\mathbf{v}}^{*}+{\dot{\mathbf{e}}}_i\right|}^2\right){\dot{\mathbf{x}}}_i\hfill \\ =a\left[1-\left({\mathbf{v}}^{*}{\mathbf{v}}^{*}+2{\mathbf{v}}^{*}{\dot{\mathbf{e}}}_i+{{\dot{\mathbf{e}}}_i}^3\right)\right]{\dot{\mathbf{x}}}_i\hfill \\ =-a\left(2{\mathbf{v}}^{*}{\dot{\mathbf{e}}}_i+{{\dot{\mathbf{e}}}_i}^2\right)\left({\mathbf{v}}^{*}+{\dot{\mathbf{e}}}_i\right)\hfill \\ =-a\left(2{\mathbf{v}}^{*}{\dot{\mathbf{e}}}_i·{\mathbf{v}}^{*}+{{\dot{\mathbf{e}}}_i}^2·{\mathbf{v}}^{*}+2{\mathbf{v}}^{*}{\dot{\mathbf{e}}}_i·{\dot{\mathbf{e}}}_i+{{\dot{\mathbf{e}}}_i}^3\right)\hfill \end{array}$$

(18)

Neglecting higher-order term $\mathcal{O}\left({{\dot{\mathbf{e}}}_i}^2\right)$, the expression simplifies to

$$a\left(1-{\left|{\dot{\mathbf{x}}}_i\right|}^2\right){\dot{\mathbf{x}}}_i=-2a{\mathbf{v}}^{*}{\dot{\mathbf{e}}}_i·{\mathbf{v}}^{*}$$

(19)

Consider the Lyapunov function candidate:

$$V\left(X\right)={\displaystyle \frac{1}{2}}\sum _{i=1}^N{\left|{\dot{\mathbf{e}}}_i\right|}^2+{\displaystyle \frac{b}{2N}}\sum _{i=1}^N\sum _{j=1,j\ne i}^{N}U\left({\mathbf{x}}_{ij}\right)$$

(20)

Its time derivative is computed:

$$\begin{array}{c}\dot{V}=\sum _{i=1}^N{\dot{\mathbf{e}}}_i·{\ddot{\mathbf{e}}}_i+{\displaystyle \frac{b}{2N}}\sum _{i=1}^N\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)·\left({\dot{\mathbf{e}}}_i-{\dot{\mathbf{e}}}_j\right)\hfill \\ =\sum _{i=1}^N{\dot{\mathbf{e}}}_i·\left[a(1-{\left|{\dot{\mathbf{x}}}_i\right|}^2){\dot{\mathbf{x}}}_i-{\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)\right]\hfill \\ +{\displaystyle \frac{b}{2N}}\sum _{i=1}^N\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)·\left({\dot{\mathbf{e}}}_i-{\dot{\mathbf{e}}}_j\right)\hfill \end{array}$$

(21)

According to Lemma 1, we can get

$${\displaystyle \frac{b}{2N}}\sum _{i=1}^N\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)·\left({\dot{\mathbf{e}}}_i-{\dot{\mathbf{e}}}_j\right)={\displaystyle \frac{b}{N}}\sum _{i=1}^N\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)·{\dot{\mathbf{e}}}_i$$

(22)

Therefore, employing (19),

$$\begin{array}{c}\dot{V}=\sum _{i=1}^N{\dot{\mathbf{e}}}_i·\left[-2a{\mathbf{v}}^{*}{\dot{\mathbf{e}}}_i·{\mathbf{v}}^{*}-{\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)\right]\hfill \\ +{\displaystyle \frac{b}{N}}\sum _{i=1}^N\sum _{j=1,j\ne i}^N\nabla U\left({\mathbf{x}}_{ij}\right)·{\dot{\mathbf{e}}}_i\hfill \\ =-2a{\left({\mathbf{v}}^{*}{\dot{\mathbf{e}}}_i\right)}^2\le 0\hfill \end{array}$$

(23)

This expression is negative semi-definite, as it is zero whenever the velocity error ${\dot{\mathbf{e}}}_i$ is orthogonal to the common direction of motion. The system is stable in the sense of Lyapunov.

To establish asymptotic stability, we invoke LaSalle’s invariance principle. The set where $\dot{V}=0$ is given by $L=\left\{\left({\mathbf{e}}_i,{\dot{\mathbf{e}}}_i\right)|{\mathbf{v}}^{*}·{\dot{\mathbf{e}}}_i=0\right\}$.

According to LaSalle’s principle, system trajectories converge to the largest invariant subset of $L=\left\{\left({\mathbf{e}}_i,{\dot{\mathbf{e}}}_i\right)|{\mathbf{v}}^{*}·{\dot{\mathbf{e}}}_i=0\right\}$, in which the velocity error ${\dot{\mathbf{e}}}_i$ is orthogonal to the collective motion direction ${\mathbf{v}}^{*}$. We now demonstrate that the only possible solution within this invariant set is ${\dot{\mathbf{e}}}_i=0$.

Within the invariant set $L=\left\{\left({\mathbf{e}}_i,{\dot{\mathbf{e}}}_i\right)|{\mathbf{v}}^{*}·{\dot{\mathbf{e}}}_i=0\right\}$, the condition ${\mathbf{v}}^{*}·{\dot{\mathbf{e}}}_i=0$ holds for all agents. Substituting this into (21), it holds that

$$\begin{array}{c}\dot{V}=\sum _{i=1}^N{\dot{\mathbf{e}}}_i·a\left(1-{\left|{\dot{\mathbf{x}}}_i\right|}^2\right){\dot{\mathbf{x}}}_i\hfill \\ =-a\sum _{i=1}^N{\dot{\mathbf{e}}}_i·\left(2{\mathbf{v}}^{*}{\dot{\mathbf{e}}}_i+{{\dot{\mathbf{e}}}_i}^2\right)\left({\mathbf{v}}^{*}+{\dot{\mathbf{e}}}_i\right)\hfill \\ =-a\sum _{i=1}^N{\left|{\dot{\mathbf{e}}}_i\right|}^4\le 0\hfill \end{array}$$

(24)

This implies that $\dot{V}=0$ if and only if $\left|{\dot{\mathbf{e}}}_i\right|=0$ for all i. Thus, the largest invariant set is $L_{max}=\left\{\left({\mathbf{e}}_i,{\dot{\mathbf{e}}}_i\right)|\left|{\dot{\mathbf{e}}}_i\right|=0\right\}$. By LaSalle’s invariance principle, all trajectories of the system converge to $L_{max}=\left\{\left({\mathbf{e}}_i,{\dot{\mathbf{e}}}_i\right)|\left|{\dot{\mathbf{e}}}_i\right|=0\right\}$, meaning that velocity errors asymptotically decay to zero. Therefore, the translational phase is locally asymptotically stable near the equilibrium. □

To prove the stability of the vortex motion phase, we make the assumption that all individuals have approximately the same angular velocity $\omega$ in the vortex motion phase.

Theorem 2. 

Governed by the phase transition control law (5), The system is stable (but not necessarily asymptotically stable) in the vicinity of the vortex phase solution defined by (13) if the control parameter $\lambda =0$ and the Hessian matrix ${{\nabla }_{\mathbf{q}}}^{2}U\left({\mathbf{r}}_{ij}\right)={\mathbf{H}}_{ij}^q$ satisfy ${\mathbf{JH}}_{ij}^q={\mathbf{H}}_{ij}^q\mathbf{J}$, where $\mathbf{J}=\left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)$.

Proof. 

Based on the results in Section 3, when $\mathbf{R}={\mathbf{R}}_0$, i.e., $\dot{\mathbf{R}}=0$, each agent exhibits a periodic motion around the center of mass. The stability analysis for the vortex phase is conducted in a rotating coordinate frame that co-moves with the desired collective rotation. Furthermore, since the center of mass remains stationary in the vortex solution, a coordinate system is established with the center of mass as the origin.

Within the vortex phase, let $\mathbf{J}$ be the rotation matrix, and ${{\mathbf{x}}_i}^{*}=e^{\omega t\mathbf{J}}{\mathbf{r}}_i$ be the desired position, where ${\mathbf{r}}_i$ is a constant vector. The actual position of agent i is denoted as ${\mathbf{x}}_i=e^{\omega \mathbf{J}t}{\mathbf{q}}_i$, where ${\mathbf{q}}_i={\mathbf{r}}_i+{\mathit{\xi }}_i$, ${\mathit{\xi }}_i$ is the position error. All subsequent analysis for the vortex phase will be conducted using the above notations.

According to matrix function differentiation rules, the velocity ${\dot{\mathbf{x}}}_i$ and acceleration ${\ddot{\mathbf{x}}}_i$ of each agent can be expressed as

$${\dot{\mathbf{x}}}_i=e^{\omega \mathbf{J}t}\left({\dot{\mathbf{q}}}_i+\omega \mathbf{J}{\mathbf{q}}_i\right)$$

(25)

$${\ddot{\mathbf{x}}}_i=e^{\omega \mathbf{J}t}\left({\ddot{\mathbf{q}}}_i+2\omega \mathbf{J}{\dot{\mathbf{q}}}_i+{\omega }^2{\mathbf{J}}^2{\mathbf{q}}_i\right)=e^{\omega \mathbf{J}t}\left({\ddot{\mathbf{q}}}_i+2\omega \mathbf{J}{\dot{\mathbf{q}}}_i-{\omega }^2{\mathbf{q}}_i\right)$$

(26)

Considering that the potential gradient depends only on the distance between agents, $U\left({\mathbf{x}}_{ij}\right)=f\left(\left|{\mathbf{x}}_{ij}\right|\right)$, and $\left|e^{\omega \mathbf{J}t}\right|=1$, it holds that

$$\begin{array}{cc}\hfill \nabla U\left(e^{\omega t\mathbf{J}}{\mathbf{q}}_{ij}\right)& =f^{\prime }\left(\left|e^{\omega \mathbf{J}t}{\mathbf{q}}_{ij}\right|\right){\displaystyle \frac{e^{\omega \mathbf{J}t}{\mathbf{q}}_{ij}}{\left|e^{\omega \mathbf{J}t}{\mathbf{q}}_{ij}\right|}}=f^{\prime }\left(\left|{\mathbf{q}}_{ij}\right|\right){\displaystyle \frac{e^{\omega \mathbf{J}t}{\mathbf{q}}_{ij}}{\left|{\mathbf{q}}_{ij}\right|}}\hfill \\ & =e^{\omega \mathbf{J}t}{f}^{\prime }\left(\left|{\mathbf{q}}_{ij}\right|\right){\displaystyle \frac{{\mathbf{q}}_{ij}}{\left|{\mathbf{q}}_{ij}\right|}}=e^{\omega \mathbf{J}t}{\nabla }_{\mathbf{q}}U\left({\mathbf{q}}_{ij}\right)\hfill \end{array}$$

(27)

Substituting the expression into the system dynamics (4), we have

$$\begin{array}{cc}& e^{\omega \mathbf{J}t}\left({\ddot{\mathbf{q}}}_i+2\omega \mathbf{J}{\dot{\mathbf{q}}}_i-{\omega }^2{\mathbf{q}}_i\right)\hfill \\ \hfill =& a\left(1-{\left|e^{\omega \mathbf{J}t}\left({\dot{\mathbf{q}}}_i+\omega \mathbf{J}{\mathbf{q}}_i\right)\right|}^2\right)\left[e^{\omega \mathbf{J}t}\left({\dot{\mathbf{q}}}_i+\omega \mathbf{J}{\mathbf{q}}_i\right)\right]-{\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N{e}^{\omega \mathbf{J}t}{\nabla }_{\mathbf{q}}U\left({\mathbf{q}}_{ij}\right).\hfill \end{array}$$

(28)

Multiplying both sides by $e^{\omega \mathbf{J}t}$, we obtain an equation governing the position error in the vortex phase:

$${\ddot{\mathbf{q}}}_i+2\omega \mathbf{J}{\dot{\mathbf{q}}}_i-{\omega }^2{\mathbf{q}}_i=a\left(1-{\left|\left({\dot{\mathbf{q}}}_i+\omega \mathbf{J}{\mathbf{q}}_i\right)\right|}^2\right)\left({\dot{\mathbf{q}}}_i+\omega \mathbf{J}{\mathbf{q}}_i\right)-{\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N{\nabla }_{\mathbf{q}}U\left({\mathbf{q}}_{ij}\right)$$

(29)

It is noted that the new gradient ${\nabla }_{\mathbf{q}}U\left(·\right)$ shares the same form as the standard gradient ${\nabla }_{\mathbf{r}}U\left(·\right)$. The subscript merely serves to specify the coordinate and to distinguish it from the ordinary gradient operator. Henceforth in the vortex phase analysis, all derivatives will be taken with respect to the q-coordinates.

Specifically, substituting the equilibrium system state (13) into (29), we can obtain

$${\omega }^2{\mathbf{r}}_i={\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N{\nabla }_{\mathbf{q}}U\left({\mathbf{r}}_{ij}\right)$$

(30)

This equation is demonstrated to admit a solution, which can be verified by integrating over an appropriate domain, such as a circle, leading to the specific equation for each individual agent.

Define the following Lyapunov function candidate

$$V={\displaystyle \frac{1}{2}}\sum _{i=1}^N{\left|{\mathit{\eta }}_i+\omega \mathbf{J}{\mathit{\xi }}_i\right|}^2+\Phi \left(\left\{{\mathbf{q}}_i\right\}\right)-\Phi \left(\left\{{\mathbf{r}}_i\right\}\right)+{\displaystyle \frac{1}{2}}\sum _{i=1}^N{\left|\omega {\mathit{\xi }}_i\right|}^2$$

(31)

where $\Phi \left(\left\{{\mathbf{q}}_i\right\}\right)={\displaystyle \frac{b}{2N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{j=1,j\ne i}^N}U\left({\mathbf{q}}_{ij}\right)-{\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^N}{\omega }^2{\left|{\mathbf{q}}_i\right|}^2$ is the effective potential energy function.

The proof begins by demonstrating the positive definiteness of this Lyapunov function. By expending the effective potential energy term, we get

$$\begin{gathered} \begin{array}{c}\Phi \left(\left\{{\mathbf{q}}_i\right\}\right)-\Phi \left(\left\{{\mathbf{r}}_i\right\}\right)={\displaystyle \frac{b}{2N}}\sum _{i=1}^N\left(\sum _{j=1,j\ne i}^{N}U\left({\mathbf{q}}_{ij}\right)-\sum _{j=1,j\ne i}^{N}U\left({\mathbf{r}}_{ij}\right)\right)-{\displaystyle \frac{1}{2}}\left(\sum _{i=1}^N{\omega }^2{\left|{\mathbf{q}}_i\right|}^2-\sum _{i=1}^N{\omega }^2{\left|{\mathbf{r}}_i\right|}^2\right) \\ ={\displaystyle \frac{b}{2N}}\sum _{i=1}^N\left(\sum _{j=1,j\ne i}^{N}U\left({\mathbf{r}}_{ij}+{\mathit{\xi }}_{ij}\right)-\sum _{j=1,j\ne i}^{N}U\left({\mathbf{r}}_{ij}\right)\right)-{\displaystyle \frac{1}{2}}\left(2\sum _{i=1}^N{\omega }^2{\mathbf{r}}_i·{\mathit{\xi }}_i+\sum _{i=1}^N{\omega }^2{{\mathit{\xi }}_i}^2\right)\hfill \end{array} \end{gathered}$$

(32)

Consequently, the complete Lyapunov function can be expressed as follows:

$$V={\displaystyle \frac{1}{2}}\sum _{i=1}^N{\left|{\mathit{\eta }}_i+\omega \mathbf{J}{\mathit{\xi }}_i\right|}^2+{\displaystyle \frac{b}{2N}}\sum _{i=1}^N\left(\sum _{j=1,j\ne i}^{N}U\left({\mathbf{r}}_{ij}+{\mathit{\xi }}_{ij}\right)-\sum _{j=1,j\ne i}^{N}U\left({\mathbf{r}}_{ij}\right)\right)-\sum _{i=1}^N{\omega }^2{\mathbf{r}}_i·{\mathit{\xi }}_i$$

(33)

Expand (31) based on error:

$$\begin{array}{cc}& {\displaystyle \frac{b}{2N}}\sum _{i=1}^N\left(\sum _{j=1,j\ne i}^{N}U\left({\mathbf{r}}_{ij}+{\mathit{\xi }}_{ij}\right)-\sum _{j=1,j\ne i}^{N}U\left({\mathbf{r}}_{ij}\right)\right)\hfill \\ \hfill =& {\displaystyle \frac{b}{2N}}\sum _{i=1}^N\left(\sum _{j=1,j\ne i}^N{\nabla }_{\mathbf{q}}U\left({\mathbf{r}}_{ij}\right)·{\mathit{\xi }}_{ij}+{\displaystyle \frac{1}{2}}\sum _{j=1,j\ne i}^N{{\mathit{\xi }}_{ij}}^T{{\nabla }_{\mathbf{q}}}^{2}U\left({\mathbf{r}}_{ij}\right){\mathit{\xi }}_{ij}+\mathcal{O}\left({\left|{\mathit{\xi }}_{ij}\right|}^2\right)\right)\hfill \end{array}$$

(34)

Substituting (30) into (33), it holds that

$${\displaystyle \frac{b}{2N}}\sum _{j=1,j\ne i}^N{\nabla }_{\mathbf{q}}U\left({\mathbf{r}}_{ij}\right)·{\mathit{\xi }}_{ij}={\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N{\nabla }_{\mathbf{q}}U\left({\mathbf{r}}_{ij}\right)·{\mathit{\xi }}_i={\omega }^2{\mathbf{r}}_i·{\mathit{\xi }}_i$$

(35)

The linearized Lyapunov function can be expressed as

$$V={\displaystyle \frac{1}{2}}\sum _{i=1}^N{\left|{\mathit{\eta }}_i+\omega \mathbf{J}{\mathit{\xi }}_i\right|}^2+{\displaystyle \frac{b}{4}}\sum _{i=1}^N\sum _{j=1,j\ne i}^N{{\mathit{\xi }}_{ij}}^T{{\nabla }_{\mathbf{q}}}^{2}U\left({\mathbf{r}}_{ij}\right){\mathit{\xi }}_{ij}$$

(36)

Based on the positive definite assumption in Equation (3), there exists ${{\mathit{\xi }}_{ij}}^T{{\nabla }_{\mathbf{q}}}^{2}U\left({\mathbf{r}}_{ij}\right){\mathit{\xi }}_{ij}={{\mathit{\xi }}_{ij}}^T{\mathbf{H}}_{ij}^q{\mathit{\xi }}_{ij}\ge 0$. Therefore, the Lyapunov function is always greater than or equal to zero around the vortex phase, with equality holding only when both the position error and its derivative are simultaneously zero. This completes the proof of positive definiteness.

Taking the time derivative of the Lyapunov function (33), we can get

$$\dot{V}=\sum _{i=1}^N{\left({\mathit{\eta }}_i+\omega \mathbf{J}{\mathit{\xi }}_i\right)}^T\left({\dot{\mathit{\eta }}}_i+\omega \mathbf{J}{\dot{\mathit{\xi }}}_i\right)+{\displaystyle \frac{b}{2N}}\sum _{i=1}^N\sum _{j=1,j\ne i}^N{\nabla }_{\mathbf{q}}U\left({\mathbf{r}}_{ij}+{\mathit{\xi }}_{ij}\right){\dot{\mathit{\xi }}}_{ij}-\sum _{i=1}^N{\omega }^2{\mathbf{r}}_i·{\dot{\mathit{\xi }}}_i$$

(37)

Examining the first term of the time derivative in (37), and noting ${\dot{\mathbf{q}}}_i+\omega \mathbf{J}{\mathbf{q}}_i={\dot{\mathit{\xi }}}_i+\omega \mathbf{J}\left({\mathbf{r}}_i+{\mathit{\xi }}_i\right)=\omega \mathbf{J}{\mathbf{r}}_i+\left({\mathit{\eta }}_i+\omega \mathbf{J}{\mathit{\xi }}_i\right)$, let ${\mathbf{u}}_i={\mathit{\eta }}_i+\omega \mathbf{J}{\mathit{\xi }}_i$ to simplify the expression, the dynamics of error (30) can be written as

$${\dot{\mathit{\eta }}}_i+2\omega \mathbf{J}{\dot{\mathit{\xi }}}_i-{\omega }^2\left({\mathbf{r}}_i+{\mathit{\xi }}_i\right)=a\left(1-{\left|\omega \mathbf{J}{\mathbf{r}}_i+{\mathbf{u}}_i\right|}^2\right)\left(\omega \mathbf{J}{\mathbf{r}}_i+{\mathbf{u}}_i\right)-{\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N{\nabla }_{\mathbf{q}}U\left({\mathbf{r}}_{ij}+{\mathit{\xi }}_{ij}\right)$$

(38)

Substituting (38) into the first term of Equation (37), we can obtain

$$\begin{array}{cc}\hfill & \sum _{i=1}^N{\left({\mathit{\eta }}_i+\omega \mathbf{J}{\mathit{\xi }}_i\right)}^T\left({\dot{\mathit{\eta }}}_i+\omega \mathbf{J}{\dot{\mathit{\xi }}}_i\right)\hfill \\ \hfill & =a\sum _{i=1}^N{{\mathbf{u}}_i}^T\left(1-{\left|\omega \mathbf{J}{\mathbf{r}}_i+{\mathbf{u}}_i\right|}^2\right)\left(\omega \mathbf{J}{\mathbf{r}}_i+{\mathbf{u}}_i\right)+\sum _{i=1}^N{\left({\mathit{\eta }}_i+\omega \mathbf{J}{\mathit{\xi }}_i\right)}^T\left({\omega }^2{\mathit{\xi }}_i-\omega \mathbf{J}{\dot{\mathit{\xi }}}_i\right)\hfill \\ \hfill & -\sum _{i=1}^N{\left({\mathit{\eta }}_i+\omega \mathbf{J}{\mathit{\xi }}_i\right)}^T\left[{\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N{\nabla }_{\mathbf{q}}U\left({\mathbf{r}}_{ij}+{\mathit{\xi }}_{ij}\right)-{\omega }^2{\mathbf{r}}_i\right]\hfill \end{array}$$

(39)

Leveraging the skew-symmetric property of the matrix ${\mathbf{J}}^T\mathbf{J}=\mathbf{I}$, and for any vector $\mathbf{a}$, it holds that ${\mathbf{a}}^T\mathbf{Ja}=0$, therefore

$$\begin{array}{cc}\hfill & \sum _{i=1}^N{\left({\mathit{\eta }}_i+\omega \mathbf{J}{\mathit{\xi }}_i\right)}^T\left({\omega }^2{\mathit{\xi }}_i-\omega \mathbf{J}{\dot{\mathit{\xi }}}_i\right)\hfill \\ \hfill & =\sum _{i=1}^N{\omega }^2{{\mathit{\eta }}_i}^T{\mathit{\xi }}_i+{\omega }^2{\left(\omega \mathbf{J}{\mathit{\xi }}_i\right)}^T{\mathit{\xi }}_i-{{\mathit{\eta }}_i}^T\left(\omega \mathbf{J}{\dot{\mathit{\xi }}}_i\right)-{\left(\omega \mathbf{J}{\mathit{\xi }}_i\right)}^T\left(\omega \mathbf{J}{\dot{\mathit{\xi }}}_i\right)\hfill \\ \hfill & =\sum _{i=1}^N{\omega }^2{{\mathit{\eta }}_i}^T{\mathit{\xi }}_i-{\left(\omega \mathbf{J}{\mathit{\xi }}_i\right)}^T\left(\omega \mathbf{J}{\dot{\mathit{\xi }}}_i\right)=0\hfill \end{array}$$

(40)

According to Lemma 1, it holds that

$${\displaystyle \frac{b}{N}}\sum _{i=1}^N{{\mathit{\eta }}_i}^T\sum _{j=1,j\ne i}^N{\nabla }_{\mathbf{q}}U\left({\mathbf{r}}_{ij}+{\mathit{\xi }}_{ij}\right)={\displaystyle \frac{b}{2N}}\sum _{i=1}^N\sum _{j=1,j\ne i}^N{\nabla }_{\mathbf{q}}U\left({\mathbf{r}}_{ij}+{\mathit{\xi }}_{ij}\right)·{\dot{\mathit{\xi }}}_{ij}$$

(41)

Substituting (41) into (37), the final expression for the time derivative of the Lyapunov function is obtained as

$$\dot{V}=a\sum _{i=1}^N{{\mathbf{u}}_i}^T\left(1-{\left|\omega \mathbf{J}{\mathbf{r}}_i+{\mathbf{u}}_i\right|}^2\right)\left(\omega \mathbf{J}{\mathbf{r}}_i+{\mathbf{u}}_i\right)-\sum _{i=1}^N{\left(\omega \mathbf{J}{\mathit{\xi }}_i\right)}^T\left[{\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N{\nabla }_{\mathbf{q}}U\left({\mathbf{r}}_{ij}+{\mathit{\xi }}_{ij}\right)-{\omega }^2{\mathbf{r}}_i\right]$$

(42)

To analyze its definiteness, this derivative is expanded around the equilibrium point.

For the first term of the time derivative,

$$\begin{array}{cc}\hfill & \left[1-{\left(\omega \mathbf{J}{\mathbf{r}}_i+{\mathbf{u}}_i\right)}^2\right]\left(\omega \mathbf{J}{\mathbf{r}}_i+{\mathbf{u}}_i\right)\hfill \\ \hfill & =-\left[2\omega {\left(\mathbf{J}{\mathbf{r}}_i\right)}^T{\mathbf{u}}_i+{{\mathbf{u}}_i}^2\right]\left(\omega \mathbf{J}{\mathbf{r}}_i+{\mathbf{u}}_i\right)\hfill \\ \hfill & =-2{\omega }^2{\left(\mathbf{J}{\mathbf{r}}_i\right)}^T{\mathbf{u}}_i\left(\mathbf{J}{\mathbf{r}}_i\right)+\mathcal{O}\left({\left|{\mathbf{u}}_i\right|}^2\right)\hfill \end{array}$$

(43)

For the second term

$$\sum _{i=1}^N{\left(\omega \mathbf{J}{\mathit{\xi }}_i\right)}^T\left[{\displaystyle \frac{b}{N}}\sum _{j=1,j\ne i}^N{\nabla }_{\mathbf{q}}U\left({\mathbf{r}}_{ij}+{\mathit{\xi }}_{ij}\right)-{\omega }^2{\mathbf{r}}_i\right]={\displaystyle \frac{b}{N}}\sum _{i=1}^N{\left(\omega \mathbf{J}{\mathit{\xi }}_i\right)}^T\left(\sum _{j=1,j\ne i}^N{{\nabla }_{\mathbf{q}}}^{2}U\left({\mathbf{r}}_{ij}\right){\mathit{\xi }}_{ij}\right)$$

(44)

Let ${{\nabla }_{\mathbf{q}}}^{2}U\left({\mathbf{r}}_{ij}\right)={\mathbf{H}}_{ij}^q$, and according to the isotropy assumption of the potential function ${\mathbf{H}}_{ij}^q={\mathbf{H}}_{ji}^q$, we can get

$$\sum _{i=1}^N\omega \sum _{j=1,j\ne i}^N{{\mathit{\xi }}_i}^T{\mathbf{J}}^T{\mathbf{H}}_{ij}^q{\mathit{\xi }}_{ij}=\sum _{j=1,j\ne i}^N\omega \sum _{i=1}^N{{\mathit{\xi }}_j}^T{\mathbf{J}}^T{\mathbf{H}}_{ji}^q{\mathit{\xi }}_{ji}=-\sum _{j=1,j\ne i}^N\omega \sum _{i=1}^N{{\mathit{\xi }}_j}^T{\mathbf{J}}^T{\mathbf{H}}_{ij}^q{\mathit{\xi }}_{ij}$$

(45)

Therefore

$$\begin{array}{cc}\hfill \sum _{i=1}^N\omega \sum _{j=1,j\ne i}^N{{\mathit{\xi }}_i}^T{\mathbf{J}}^T{\mathbf{H}}_{ij}^q{\mathit{\xi }}_{ij}& ={\displaystyle \frac{1}{2}}\left(\sum _{i=1}^N\omega \sum _{j=1,j\ne i}^N{{\mathit{\xi }}_i}^T{\mathbf{J}}^T{\mathbf{H}}_{ij}^q{\mathit{\xi }}_{ij}-\sum _{j=1}^N\omega \sum _{i=1,i\ne j}^N{{\mathit{\xi }}_j}^T{\mathbf{J}}^T{\mathbf{H}}_{ij}^q{\mathit{\xi }}_{ij}\right)\hfill \\ & ={\displaystyle \frac{1}{2}}\sum _{i=1}^N\omega \sum _{j=1,j\ne i}^N{{\mathit{\xi }}_{ij}}^T{\mathbf{J}}^T{\mathbf{H}}_{ij}^q{\mathit{\xi }}_{ij}\hfill \end{array}$$

(46)

Transposing ${{\mathit{\xi }}_{ij}}^T{\mathbf{J}}^T{\mathbf{H}}_{ij}^q{\mathit{\xi }}_{ij}$ yields

$${{\mathit{\xi }}_{ij}}^T{\mathbf{JH}}_{ij}^q{\mathit{\xi }}_{ij}=-{{\mathit{\xi }}_{ij}}^T{\mathbf{H}}_{ij}^q\mathbf{J}{\mathit{\xi }}_{ij}$$

(47)

According to condition ${\mathbf{JH}}_{ij}^q={\mathbf{H}}_{ij}^q\mathbf{J}$, it holds

$$\sum _{i=1}^N\omega \sum _{j=1,j\ne i}^N{{\mathit{\xi }}_i}^T{\mathbf{J}}^T{\mathbf{H}}_{ij}^q{\mathit{\xi }}_{ij}=0$$

(48)

Substituting (48) into (42), we finally get

$$\dot{V}=-2a\sum _{i=1}^N{\omega }^2{\left({{\mathbf{r}}_i}^T{\mathbf{J}}^T{\mathbf{u}}_i\right)}^2\le 0$$

(49)

Since the Lyapunov function is positive definite and its time derivative is negative semidefinite, the vortex-phase equilibrium is stable in the sense of Lyapunov. However, the derivative $\dot{V}$ does not become strictly negative definite in the entire error state space because it vanishes whenever the error ${\mathbf{u}}_i$ is perpendicular to the radial direction ${{\mathbf{r}}_i}^T{\mathbf{J}}^T$ for all i. Thus, the equilibrium is not proven to be asymptotically stable, leading to the fluctuation of the vortex motion phase in simulation. □

Remark 2. 

This condition ${\mathbf{JH}}_{ij}^q={\mathbf{H}}_{ij}^q\mathbf{J}$ ensures the cross-term (48) vanishes. The quadratic potential $U\left(r\right)={\displaystyle \frac{1}{2}}{(r-d_s)}^2$ is a trivial example satisfying this condition, whose Hessian matrix satisfies ${\mathbf{H}}_{ij}^q=\mathbf{I}$. While this condition is sufficient for proving stability in the Lyapunov sense in the vortex motion phase discussed in this study, it may not be necessary. The vortex phase observed in simulations with other potentials suggests that stability can arise even when this commutation condition does not strictly hold. Exploring a more general stability criterion for the vortex phase remains an open and challenging problem worthy of future theoretical work.

5. Simulation Result and Discussion

In this section, numerical simulations are conducted to prove the main results given in the paper. In the simulations, $N=25$ UAVs are initialized in an area of 5 × 5, with the simulation interval set as $dt=0.01$ s. In the simulations, quadratic potential function is used as an example. In particular, there has $U\left({\mathbf{x}}_{ij}\right)={\displaystyle \frac{1}{2}}{\left(\left|{\mathbf{x}}_{ij}\right|-d_s\right)}^2$.

To evaluate the stability of the two motion phases, two widely adopted order parameters are employed. The translational motion phase can be characterized by the velocity alignment order parameter $V_m$, which is calculated by the velocity alignment within the swarm, $V_m={\displaystyle \frac{1}{N}}\left|{\displaystyle \sum _{i=1}^N}{\displaystyle \frac{{\dot{\mathbf{x}}}_i}{\left|{\dot{\mathbf{x}}}_i\right|}}\right|$. In translational motion phase, $V_m=1$. Conversely, the vortex motion phase is captured by the rotational order parameter $V_c={\displaystyle \frac{1}{N}}\left|{\displaystyle \sum _{i\in N_i}}{\displaystyle \frac{{\mathbf{r}}_i\times {\dot{\mathbf{x}}}_i}{\left|{\mathbf{r}}_i\right|\left|{\dot{\mathbf{x}}}_i\right|}}\right|$, where $N_i$ indicates the distance neighbors of the individual i (not necessarily the communication neighbor of the individual), ${\mathbf{r}}_i$ indicates the center of the vortex center. The rotational order parameter measures the consistency of circular motion around a common center. If $V_c$ is close to 1, the swarm is considered to be in the vortex motion phase.

5.1. Stability of the Vortex Motion Phase

To validate the existence and stability of the vortex phase, the parameters used in the simulation are $a=0.2$, $b=5$, $\lambda =0$, $d_s=0.5$, and all UAVs are randomly distributed in an area of 5 × 5. Under these conditions, the swarm consistently converges to a vortex motion phase, where all individuals approximately move around a center, as shown in Figure 3.

The stability of this vortex phase is verified by examining the rotational order parameter $V_c$, as shown in Figure 4. It is observed that the rotational order parameter evolves from an initial disordered state and converges to a value near 1, demonstrating the swarm’s ability to establish and maintain the vortex phase. The persistent, small-amplitude oscillations in the steady-state indicate that the vortex phase exhibits stability, while asymptotic stability is generally not attained.

Furthermore, the individual angular velocities of all UAVs with respect to the swarm’s center of mass are computed throughout the process, as shown in Figure 5 (an enlarged view is provided for clarity). The results show that the angular velocities of all drones converge to approximately the same value, validating the uniform angular velocity assumption employed in the vortex stability analysis.

5.2. Stability of the Translational Motion Phase

The translational motion phase is gained through a transition from the vortex motion phase. To gain a stable translational motion phase, the initial positions and velocities of the swarm are set to be the same as the final position of the vortex phase, and a simple roosting force ${\mathbf{F}}_i^{roost}({\mathbf{x}}_i,{\mathbf{v}}_i)={\left[1,0\right]}^T$ is introduced just to trigger the transition. In the beginning of the transition process, the roosting force term is activated with $\lambda =0.02$. The swarm is allowed to evolve under the influence of the complete control law until the speed of the center of mass exceeds a threshold of $0.7$. Subsequently, the roosting force term is disabled with $\lambda =0$ to observe the swarm’s motion phase. The final collective state, depicted in Figure 6, confirms a transition to a coherent translational formation.

The diagram of the order parameters in the transition process provides further insight into the stability of this motion phase, as shown in Figure 7. It is observed that despite the absence of the roosting force in the latter stage, the swarm successfully converges from the vortex phase to a stable translational motion phase. Notably, the translational order parameter $V_m$ converges to a value near 1 and, in contrast to the oscillatory behavior of $V_c$ in the vortex phase, exhibits negligible fluctuations. This aligns with the local asymptotic stability property proven for the translational phase in Theorem 1, indicating a higher degree of robustness in this motion phase.

Figure 8 displays the velocity error for all UAVs within the swarm during the transition process. The error for the i-th UAV is defined as the magnitude of its velocity relative to the swarm’s center of mass, calculated as $\left|{\dot{\mathbf{x}}}_i-\dot{\mathbf{R}}\right|$. The figure initially shows a clear velocity error corresponding to the vortex phase. Subsequently, the action of the roosting force rapidly reduces this collective error. Once the speed of the center of mass reaches the predefined threshold 0.7, the roosting force term is deactivated, after which the swarm’s velocity error continues to decrease, converging to near zero within approximately 50 s. This convergence persists stably for the remainder of the simulation. The decay of relative velocities to zero provides numerical validation for the theoretical stability of the translational phase. It confirms that all UAVs successfully synchronize their velocities with the group’s center of mass, achieving cohesive and stable translational motion.

Remark 3. 

The triggering condition based on the center-of-mass velocity $V_R$ is introduced here to clearly illustrate the phase transition behavior in simulations. It should be noted that this global quantity is not required for practical implementation. In distributed UAV swarm systems, $V_R$ can be replaced by locally available information, such as neighborhood velocity averages or consensus-based estimators. Moreover, as to be demonstrated in Section 5.3, the phase transition can still be achieved using purely local control laws without relying on any global velocity-related term.

5.3. Transition Between Two Motion Phases

To further demonstrate the distributed nature of the proposed framework, we next consider an alternative roosting force formulation that does not rely on any global information. The roosting force term is introduced into the dynamics inspired by the principles discussed in literature [43]. The roosting force term is designed to attract UAVs towards a designed nest location (the origin) by exerting a steer force proportion to the UAV-roost distance, giving the individuals a tendency towards the origin. The general formulation of the roosting force is

$${\mathbf{G}}_i({\mathbf{x}}_i,{\mathbf{v}}_i)=-[{\mathbf{v}}_i^{\perp }·{\nabla }_x\varphi \left({\mathbf{x}}_i\right)]{\mathbf{v}}_i^{\perp }$$

(50)

where $\varphi$ is the roosting potential function with a generic example $\varphi \left(x_i\right)={\displaystyle \frac{1}{3}}{\left|x_i\right|}^3$. This roosting force effectively converts the component of the potential gradient perpendicular to the UAV’s heading into a steering correction, guiding the UAV along potential contours towards regions of lower potential.

The initial formulation implies that the homing force increases without bound as the distance to the nest grows, which could lead to numerical instability. To address this, we introduce a saturation mechanism. Therefore, the final roosting force term is saturated:

$${\mathbf{F}}_i^{roost}({\mathbf{x}}_i,{\mathbf{v}}_i)={\displaystyle \frac{{\mathbf{G}}_i({\mathbf{x}}_i,{\mathbf{v}}_i)}{max\left\{1,\left|{\mathbf{G}}_i({\mathbf{x}}_i,{\mathbf{v}}_i)\right|\right\}}}$$

(51)

where $\varphi \left({\mathbf{x}}_i\right)={\displaystyle \frac{1}{3}}{\left|{\mathbf{x}}_i\right|}^3$. The saturation process limits the maximum amplitude of the steering force to 1, ensuring bounded control input and improves simulation robustness. In the simulation, the parameter $\lambda$ is set as 0.05, ensuring the roosting force is persistent but does not excessively dominate the intrinsic swarm dynamics.

In the simulation of phase transition, $N=25$ UAVs were initialized within a 5 × 5 square region at a distance from the roost. This region was centered at $\left(502.5,2.5\right)$, meaning the initial positions were uniformly distributed such that $x_i\in \left[500,505\right]$, $y_i\in \left[0,5\right]$. Their initial velocities were also randomly assigned.

The UAVs follow the phase control law (4) and (5) with the roosting force term defined in (51). The collective motion phase during the whole simulation process is shown in Figure 9, demonstrating a clear two-phase homing process.

In the first part of the simulation, the swarm is far from the nest with a large $\left|{\mathbf{x}}_{\mathbf{i}}\right|$, the roosting force term consistently provides a steering component, directing the UAVs towards the roost. The swarm consequently moves as a cohesive, polarized group on a towards the nest location, thus forming the translational motion phase.

As the swarm approaches near the nest, it undergoes a distinct phase transition process into the vortex motion phase. Near the origin, the magnitude of the roosting force diminishes. The dynamic is then dominated by the self-propulsion term and the swarm interaction term. As the roosting force is always perpendicular to an UAV’s velocity, it no longer pulls the UAV to the roost but rather continually adjusts its heading. This results in a stable vortex motion phase around the nest location. Due to the self-propulsion term trying to maintain a certain velocity, the UAVs do not converge to a static point but enter a dynamic equilibrium, orbiting the nest at a relatively fixed radius. This behavior is analogous to pigeons circling a loft before landing. The stability of this vortex is confirmed by the diagram of the order parameter, as shown in Figure 10.

5.4. Simulation Results with Morse Potential

To further demonstrate the generality of the proposed framework, additional simulations are conducted using a Morse-type interaction potential $U\left({\mathbf{x}}_{ij}\right)=C_r{e}^{-\left|{\mathbf{x}}_{ij}\right|/l_r}-C_a{e}^{-\left|{\mathbf{x}}_{ij}\right|/l_a}$. The Morse potential is widely used to model inter-agent interactions with short-range repulsion and long-range attraction. The parameters are selected as follows: $a=0.22$, $b=33$, $C_r=1$, $l_r=0.7$, $C_a=0.5$, $l_a=28$. Although the chosen parameters yield a Morse potential with a unique minimum, they do not strictly satisfy the positive definiteness condition (3) or the condition ${\mathbf{JH}}_{ij}^q={\mathbf{H}}_{ij}^q\mathbf{J}$ assumed in Theorem 2.

We first investigate whether the vortex phase can still emerge under the Morse potential with $\lambda =0$. The corresponding simulation results are shown in Figure 11.

It can be observed that, although the strict condition ${\mathbf{JH}}_{ij}^q={\mathbf{H}}_{ij}^q\mathbf{J}$ is not strictly satisfied, a vortex-like collective motion still emerges, as quantitatively evidenced by the rise and stabilization of the vortex order parameter in Figure 11b.

We further examine the phase transition behavior using the same roosting force term as in the previous simulations with $\lambda =0.0045$. With other simulation conditions being the same, the results are presented in Figure 12.

It can be clearly seen that the swarm is still able to achieve a transition from the translational motion phase to the vortex phase under the Morse potential. The transition process remains qualitatively consistent with the results obtained using the quadratic potential. Although the strict theoretical conditions are not fully met, a stable vortex motion phase still emerges, confirming that the phase transition mechanism is robust to various interaction potentials. The two order parameters during the phase transition process using the Morse potential is shown in Figure 13. It can be seen that the time required for the order parameters to stabilize is slightly longer compared to the quadratic potential, indicating a moderate effect on the convergence rate.

These results confirm that the proposed phase transition mechanism is not restricted to a specific interaction model, and remains effective even when more realistic potentials are adopted.

6. Conclusions

Inspired by the homing mechanisms of pigeon flocks, this paper establishes a phase transition control framework for UAV swarms by integrating the self-propulsion term, interaction mechanism and most importantly, the roosting force term. The main contribution of this work is the development of a unified analytical framework for understanding multiple collective motion phases within a single formulation. By adopting a centroid-based modeling perspective, theoretical analysis reveals the inherent existence of two distinct collective motion phases: a translational phase and a vortex phase. Furthermore, the role of the roosting force term was interpreted as a structured control input that can induce transitions between different motion patterns. The phase transition from translational motion to vortex motion was also successfully achieved under the Morse potential using the same roosting force mechanism, with several differences. This demonstrates that the proposed phase transition strategy is robust with respect to different interaction models and is not restricted to a specific potential function.

From an application perspective, the proposed framework can be interpreted as a model for UAV swarms as the interaction and the roosting force terms correspond to coordination strategies and task-driven control objectives. In this context, different motion phases correspond to mission-specific behaviors, including coordinated navigation and area surveillance. The ability to switch between these motion phases via a control parameter suggests a potential mechanism for flexible behavior design in UAV swarms.

Despite these promising results, several limitations remain. The current analysis assumes a fully connected interaction topology and simplified interaction laws, whereas practical UAV swarms typically operate under limited, distance-dependent, and time-varying communication networks. In addition, real-world systems are subject to communication delays, disturbances, and sensing uncertainties, which are not fully captured in the present formulation. In addition, a more rigorous robustness analysis is needed [44,45,46].

Overall, this work provides a unified perspective on collective motion and phase transitions in second-order multi-agent systems, and offers a preliminary step toward bridging theoretical swarm dynamics and practical UAV swarm control. The insights developed here may motivate further research toward more realistic, robust, and application-oriented swarm coordination strategies.