Vortex Crystal Stabilized by the Competition Between Multi-Spin and Out-of-Plane Dzyaloshinskii–Moriya Interactions

Abstract

Multiple-Q magnetic states encompass a broad class of noncollinear and noncoplanar spin textures generated by the superposition of spin density waves. In this study, we theoretically explore the emergence of vortex crystals formed by multiple-Q spin density waves on a two-dimensional triangular lattice with $D_{3\mathrm{h}}$ point group symmetry. Using simulated annealing applied to an effective spin model, we demonstrate that the synergy among the easy-plane single-ion anisotropy, the biquadratic interaction, and the out-of-plane Dzyaloshinsky–Moriya interaction defined in momentum space can give rise to a variety of double-Q and triple-Q vortex crystals. We further examine the role of easy-plane single-ion anisotropy in triple-Q vortex crystals and show that weakening the anisotropy drives topological transitions into skyrmion crystals with skyrmion numbers $\pm 1$ and $\pm 2$. The influence of an external magnetic field is also analyzed, revealing a field-induced phase transition from vortex crystals to single-Q conical spirals. These findings highlight the crucial role of out-of-plane Dzyaloshinskii–Moriya interactions in stabilizing unconventional vortex crystals, which cannot be realized in systems with purely polar or chiral symmetries.

1. Introduction

Noncollinear and noncoplanar spin textures have become a central focus in condensed matter physics, as they provide a natural platform for realizing topologically nontrivial states and the associated exotic transport and magnetoelectric phenomena. A hallmark of such textures is the topological Hall effect, in which a Hall voltage arises not from external magnetic fields or net magnetization but from the real-space Berry phase associated with finite scalar spin chirality [1,2,3,4,5,6,7]. Beyond transport signatures, such noncollinear and noncoplanar magnetic orders frequently engender multiferroic responses [8,9,10,11,12,13,14,15] and nonreciprocal transport [16,17,18,19,20,21], reflecting the intimate coupling between spin, charge, and lattice degrees of freedom. These functionalities establish noncollinear and noncoplanar spin textures as a key platform for both fundamental studies and the design of next-generation topological and multifunctional devices.

A widely studied route to generating noncollinear and noncoplanar magnetic textures is through multiple-Q magnetic states, which arise from the superposition of spin density waves with distinct ordering wave vectors [22,23]. Well-known examples include skyrmion crystals stabilized by a triple-Q superposition on a triangular lattice, as observed in MnSi [24,25,26], ${Fe}_{1-x}$${Co}_x$Si [27,28], FeGe [29,30], ${Cu}_2$O${SeO}_3$ [8,31,32], ${Gd}_2$Pd${Si}_3$ [33,34,35,36,37,38], ${Gd}_3$${Ru}_4$${Al}_{12}$ [39,40,41,42], and EuPtSi [43,44,45,46,47] and double-Q superpositions on a square lattice, as observed in Gd${Ru}_2$${Si}_2$ [48,49,50,51]. In addition, various multiple-Q states have been found, such as the meron–antimeron crystal in ${Co}_8$${Zn}_9$${Mn}_3$ [52] and the hedgehog lattices in Mn${Si}_{1-x}$${Ge}_x$ [53,54,55,56] and SrFe$O_3$ [57,58,59,60], and the vortex crystal in $Y_3$${Co}_8$${Sn}_4$ [61].

From the theoretical side, the stabilization mechanisms of noncollinear and noncoplanar spin textures have been extensively investigated [62]. In noncentrosymmetric systems, the competition between ferromagnetic exchange and the antisymmetric Dzyaloshinski–Moriya (DM) interaction [63,64], together with external magnetic fields, provides the canonical route to the formation of skyrmion crystals in chiral, polar, and $D_{2\mathrm{d}}$ ($S_4$) systems [65,66,67,68,69,70,71,72]. In centrosymmetric systems, where intrinsic DM interactions are forbidden by inversion symmetry, alternative mechanisms, such as frustration among short- and/or long-range exchange interactions [73,74,75], anisotropic symmetric exchange interactions [76,77,78,79,80,81,82], and dipolar interactions [83,84], have been shown to generate multiple-Q orderings. The interplay of these competing tendencies can stabilize a rich variety of multiple-Q orderings, ranging from skyrmion crystals to meron crystals and hedgehog lattices, even in the absence of inversion-symmetry breaking [62,85,86,87].

In this work, we focus on a less explored class of multiple-Q states—vortex crystals—on the triangular lattice in noncentrosymmetric environments distinct from chiral, polar, and $D_{2\mathrm{d}}$ ($S_4$) point-group symmetries. The essential mechanism underlying their stabilization is the cooperative interplay between multi-spin interactions and the out-of-plane DM interaction inherent to the $D_{3\mathrm{h}}$ point group symmetry [88]. By performing simulated annealing for an effective spin model, we demonstrate that the combination of easy-plane single-ion anisotropy, a momentum-resolved biquadratic interaction, and the out-of-plane DM interaction stabilizes a rich variety of double-Q and triple-Q vortex crystals. We further show that weakening the easy-plane anisotropy triggers topological transitions into skyrmion crystals with quantized skyrmion numbers $\pm 1$ and $\pm 2$, while the application of an external magnetic field drives the system into a single-Q conical spiral state. These findings underscore the pivotal role of out-of-plane DM interactions in stabilizing unconventional vortex crystals, which cannot be realized within purely polar or chiral symmetry classes, thereby opening new avenues for designing and controlling topological magnetic phases beyond conventional skyrmion-hosting systems.

The rest of this paper is organized as follows. In Section 2, we present the effective spin model on the two-dimensional triangular lattice, which incorporates the biquadratic interaction, the out-of-plane DM interaction, and the easy-plane single-ion anisotropy, and frustrated exchange interactions. We also describe the numerical approach based on simulated annealing. In Section 3, we investigate the stabilization of double-Q and triple-Q vortex crystals as functions of the model parameters. We then provide a detailed analysis of the magnetic phases appearing in the low-temperature phase diagram and examine the influence of easy-plane single-ion anisotropy and out-of-plane magnetic fields. Finally, in Section 4, we summarize our main findings.

2. Model and Method

We study a classical spin model defined on a two-dimensional triangular lattice subject to the $D_{3\mathrm{h}}$ point-group symmetry. The triangular-lattice layer is embedded in the $xy$ plane, with the lattice constant set to unity. Each magnetic site is assumed to be coordinated by ligands forming a triangular prism under $D_{3\mathrm{h}}$ symmetry [88]. Such ligand arrangements are known to occur in several van der Waals magnets [89], including transition-metal dichalcogenides [90,91] and ${Fe}_3$Ge${Te}_2$ [92,93,94]. The $D_{3\mathrm{h}}$ symmetry eliminates inversion centers on the bonds but preserves a mirror plane parallel to the triangular layer. As a consequence, the DM vectors are constrained to lie along the out-of-plane (z) direction, regardless of the bonds. This situation differs from chiral or polar systems, where the DM vectors lie in the $xy$ plane and vary with bond orientation [95,96,97].

To incorporate the out-of-plane DM interaction, the bilinear exchange Hamiltonian can be expressed as

$$\begin{array}{cc}\hfill {\mathcal{H}}^{\left(2\right)}=& -\sum _{i,j}\left[J_{ij}{\mathit{S}}_i·{\mathit{S}}_j+D_{ij}^z{({\mathit{S}}_i\times {\mathit{S}}_j)}^z\right],\hfill \end{array}$$

(1)

where ${\mathit{S}}_i=(S_i^x,S_i^y,S_i^z)$ denotes a classical spin of unit length at site i, $|{\mathit{S}}_i|=1$. The first term represents the conventional isotropic Heisenberg exchange interaction, while the second embodies the antisymmetric DM interaction with the DM vector aligned along the z axis. The latter arises microscopically from spin-dependent electron hopping processes mediated by spin–orbit coupling in the presence of broken inversion symmetry. This form contrasts with chiral or polar systems, where the DM vectors typically lie in the lattice plane and depend on the specific bond direction.

By applying a Fourier transformation, the bilinear Hamiltonian can be reformulated in momentum space as

$$\begin{array}{c}\hfill {\mathcal{H}}^{\left(2\right)}=-\sum _{\mathit{q}}\left[J_{\mathit{q}}{\mathit{S}}_{\mathit{q}}·{\mathit{S}}_{-\mathit{q}}+i{D}_{\mathit{q}}^z{\left({\mathit{S}}_{\mathit{q}}\times {\mathit{S}}_{-\mathit{q}}\right)}^z\right],\end{array}$$

(2)

where ${\mathit{S}}_{\mathit{q}}$ denotes the Fourier component of the classical localized spin ${\mathit{S}}_i$ and $\mathit{q}$ the crystal momentum. The first and second terms describe the contributions of the Heisenberg and DM interactions, respectively. Here, $J_{\mathit{q}}$ and $D_{\mathit{q}}^z$ are the Fourier transforms of the real-space coupling constants $J_{ij}$ and $D_{ij}^z$, respectively. For clarity, we restrict our attention to the effects of the out-of-plane DM interaction on magnetic ordering and therefore omit additional bond-dependent exchange terms that are allowed under $D_{3\mathrm{h}}$ point-group symmetry [98].

The momentum dependence of $J_{\mathit{q}}$ and $D_{\mathit{q}}^z$ originates from the underlying real-space exchange paths. To construct a tractable low-temperature phase diagram, we restrict the model to the dominant Fourier components in Equation (2). Specifically, we assume that the leading instabilities occur at a set of symmetry-related ordering wave vectors, $\pm {\mathit{Q}}_1=\pm (Q,0)$, $\pm {\mathit{Q}}_2=\pm (-Q/2,\sqrt{3}Q/2)$, and $\pm {\mathit{Q}}_3=\pm (-Q/2,-\sqrt{3}Q/2)$ with $Q=\pi /3$, which lie away from both $\mathit{q}=\mathit{0}$ and the Brillouin-zone boundary. These three vectors are connected by the threefold rotational symmetry of the triangular lattice, enforcing the relations $J_{{\mathit{Q}}_1}=J_{{\mathit{Q}}_2}=J_{{\mathit{Q}}_3}$ and $D_{{\mathit{Q}}_1}=D_{{\mathit{Q}}_2}=D_{{\mathit{Q}}_3}$; it is noted that $J_{{\mathit{Q}}_1}=J_{-{\mathit{Q}}_1}$, $D_{{\mathit{Q}}_1}=-D_{-{\mathit{Q}}_1}$, and so on. The momentum-space interactions are related to the real-space bilinear exchange interactions as a combination of a nearest-neighbor ferromagnetic exchange $J_1<0$ and further-neighbor antiferromagnetic couplings $J_2$ and $J_3$. For instance, the commensurate ordering wave vectors at ${\mathit{Q}}_1$–${\mathit{Q}}_3$ under periodic boundary conditions are realized by setting $J_1=1$ and $J_3=-0.394337$ with $J_2=0$. The qualitative results do not depend on the precise value of Q, as long as the ordering vectors remain away from the zone center and boundary. Under these assumptions, the bilinear Hamiltonian reduces to [82]

$$\begin{array}{c}\hfill {\mathcal{H}}^{\left(2\right)}=-\sum _{\nu }\left[J{\mathit{S}}_{{\mathit{Q}}_{\nu }}·{\mathit{S}}_{-{\mathit{Q}}_{\nu }}+iD{\left({\mathit{S}}_{{\mathit{Q}}_{\nu }}\times {\mathit{S}}_{-{\mathit{Q}}_{\nu }}\right)}^z\right],\end{array}$$

(3)

where the subscript of $J_{{\mathit{Q}}_{\nu }}$ and $D_{{\mathit{Q}}_{\nu }}$ is omitted for notational simplicity. Because of the underlying $C_3$ symmetry, spin states with spectral weight at ${\mathit{Q}}_1$, ${\mathit{Q}}_2$, or ${\mathit{Q}}_3$ are strictly degenerate. Moreover, in the absence of bond-dependent exchange interaction, the model contains no intrinsic anisotropy within the $xy$ spin plane.

Equation (3) can be interpreted as an effective spin Hamiltonian that captures the low-energy ground-state properties of the original real-space model ${\mathcal{H}}^{\left(2\right)}$ in noncentrosymmetric magnets with relativistic spin–orbit coupling. The momentum-space interactions correspond to the Fourier transforms of real-space short-range couplings, such as frustrated exchange interactions, or long-range interactions in itinerant electron systems. For the latter case, Equation (3) serves as an effective representation of long-range spin interactions mediated by conduction electrons, such as those described by the Ruderman–Kittel–Kasuya–Yosida (RKKY) mechanism [99,100,101].

The ground state of the bilinear model in Equation (3) is described by the single-Q spiral state, where the ordering wave vector is selected out of ${\mathit{Q}}_1$–${\mathit{Q}}_3$. In other words, there are no instabilities toward multiple-Q states. To promote instabilities toward multiple-Q formation in the presence of the out-of-plane DM interaction, we additionally consider three effects. One is the biquadratic interaction, which is given by

$$\begin{array}{c}\hfill {\mathcal{H}}^{\left(4\right)}={\displaystyle \frac{K}{N}}\sum _{\nu }{\left({\mathit{S}}_{{\mathit{Q}}_{\nu }}·{\mathit{S}}_{-{\mathit{Q}}_{\nu }}\right)}^2,\end{array}$$

(4)

where N is the total number of sites. This term arises from the fourth-order perturbation expansion in terms of the spin–charge coupling in the Kondo lattice model, which corresponds to the higher-order RKKY interaction [102]. We suppose that the coupling constant is positive ($K>0$) by considering the dominant contribution of the Fermi surface nesting [102]. The second is the easy-plane single-ion anisotropy, which is represented as

$$\begin{array}{c}\hfill {\mathcal{H}}^{\mathrm{SIA}}=-A\sum _i{\left(S_i^z\right)}^2,\end{array}$$

(5)

with $A<0$ favoring spin alignment in the $xy$ plane. The third is the Zeeman coupling in the presence of an external magnetic field, which is given by

$$\begin{array}{c}\hfill {\mathcal{H}}^{\mathrm{Z}}=-H\sum _i{S}_i^z,\end{array}$$

(6)

where H is the magnitude of the magnetic field. We examine the instability toward multiple-Q states for the total Hamiltonian defined as

$$\begin{array}{c}\hfill \mathcal{H}={\mathcal{H}}^{\left(2\right)}+{\mathcal{H}}^{\left(4\right)}+{\mathcal{H}}^{\mathrm{SIA}}+{\mathcal{H}}^{\mathrm{Z}},\end{array}$$

(7)

where we adopt $J=1$ as the unit of energy. In the following, we treat D, K, A, and H as phenomenological parameters. Specifically, we calculate the phase diagram in a wide range of model parameters as $0\le D\le 0.5$ and $0\le K\le 1$ to illustrate the qualitative phase behavior. While in realistic materials the bilinear exchange is usually much larger than the DM interaction and biquadratic interaction terms, our choice emphasizes the competition between these interactions. Similar parameter regimes have been suggested in itinerant magnets, e.g., ${Gd}_2$Pd${Si}_3$ [103] and MnGe [53,104], where higher-order interactions play a significant role. As shown below, we have confirmed that the main qualitative features of the phase diagrams, such as the stabilization of multiple-Q vortex crystals, are robust against moderate changes in the relative ratios of $D/J$ and $K/J$.

The effective spin model in Equation (7) can be used to search for new stabilization mechanisms of multiple-Q states and to reproduce the experimental phase diagram including multiple-Q phases [62]. In particular, by incorporating the momentum-resolved DM interaction, the effective spin model is capable of reproducing the complex transitions observed between distinct short-period skyrmion crystal phases in EuNi${Ge}_3$ [105].

To obtain the low-temperature magnetic phase diagrams for the model defined in Equation (7), we employed unbiased simulated annealing on clusters containing $N={24}^2$ spins. The simulations were performed using the standard Metropolis algorithm with local spin updates in real space. For each set of model parameters $(D,K,A,H)$, the system was initialized from a random spin configuration at high temperatures, typically chosen within the range $T_0=1$–5. The temperature was then gradually reduced according to a multiplicative cooling schedule with rates $\alpha =0.99999$–$0.999999$, until a final target temperature of $T={10}^{-4}$ was reached. At the lowest temperature, ${10}^5$–${10}^6$ Monte Carlo sweeps were performed for equilibration and subsequent measurements. To accurately identify phase boundaries, especially in regions where competing spin configurations appeared, additional simulations were initialized from low-temperature spin states previously obtained in neighboring parameter regions. This procedure allowed us to avoid metastability effects and determine the phase diagram with higher reliability.

To characterize the numerically obtained spin states, we analyze both spin-related and chirality-related quantities. In the spin sector, we evaluate the Fourier components of the magnetic moment for each spin direction $\mu =x,y,z$ at wave vector $\mathit{q}$, $m_{\mathit{q}}^{\mu }$, and the spin structure factor, $S_s^{\mu }\left(\mathit{q}\right)$, defined as

$$\begin{array}{cc}\hfill m_{\mathit{q}}^{\mu }& =\sqrt{{\displaystyle \frac{S_s^{\mu }\left(\mathit{q}\right)}{N}}},\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill S_s^{\mu }\left(\mathit{q}\right)& ={\displaystyle \frac{1}{N}}\sum _{j,l}{S}_j^{\mu }{S}_l^{\mu }{e}^{i\mathit{q}·({\mathit{r}}_j-{\mathit{r}}_l)},\hfill \end{array}$$

(9)

where ${\mathit{r}}_j$ is the real-space position of site j on the triangular lattice. For convenience, we also compute the total structure factor, $S_s\left(\mathit{q}\right)={\sum }_{\mu }{S}_s^{\mu }\left(\mathit{q}\right)$. The uniform magnetization is obtained from the site-averaged spin component $M^{\mu }=(1/N){\sum }_i{S}_i^{\mu }$.

In the chirality sector, we calculate the scalar spin chirality, which measures the noncoplanarity of three neighboring spins on each triangular plaquette. It is given by

$$\begin{array}{c}\hfill {\chi }^{\mathrm{sc}}={\displaystyle \frac{1}{N}}\sum _{\mathit{R}}{\mathit{S}}_j·\left({\mathit{S}}_k\times {\mathit{S}}_l\right),\end{array}$$

(10)

where $\mathit{R}$ denotes the triangle plaquette center, and the spins ${\mathit{S}}_j$, ${\mathit{S}}_k$, and ${\mathit{S}}_l$ are taken in counterclockwise order on upward and downward triangles. A finite value of ${\chi }^{\mathrm{sc}}$ serves as a hallmark of topologically nontrivial spin textures, such as skyrmion crystals.

3. Results

In this section, we explore how double-Q and triple-Q vortex crystals are stabilized as functions of the underlying model parameters. We begin in Section 3.1 with a detailed characterization of the magnetic phases that emerge in the low-temperature phase diagram. The role of easy-plane single-ion anisotropy is then examined in Section 3.2, followed by an analysis of the effects of an out-of-plane magnetic field in Section 3.3.

3.1. Magnetic Phase Diagram

We construct the low-temperature phase diagram of the effective spin model in Equation (7) on the triangular lattice with $D_{3\mathrm{h}}$ symmetry, focusing on the competition between the biquadratic interaction and the out-of-plane DM interaction; in this section, we fix $A=-0.4$ and $H=0$. The phase diagram, summarized in Figure 1, is presented as functions of the DM coupling constant D and the biquadratic coupling constant K. Four distinct ordered phases are stabilized: the single-Q spiral state denoted as 1Q, the double-Q vortex crystal denoted as 2Q VX, and two symmetry-inequivalent triple-Q vortex crystals denoted as 3Q VX I and 3Q VX II. All of these phases are characterized by coplanar spin configurations in the $xy$ plane, which is attributed to the strong easy-plane anisotropy. The phase diagram thus reveals that the present model can host a remarkably rich set of vortex-type spin textures, with the triple-Q vortex crystal II phase stabilized only when both D and K are finite, indicating the importance of their cooperative effect.

For small K and finite D, the system stabilizes the single-Q spiral state, which is characterized by a single ordering wave vector and a coplanar spiral configuration of spins in the $xy$ plane. This spin configuration maximizes the energy gain from the DM interaction and the easy-plane anisotropy. The spin structure factor exhibits a single pair of Bragg peaks at $\pm {\mathit{Q}}_3$ [Figure 2a], and the corresponding real-space configuration shows uniform spirals without any topological winding, as shown in Figure 3a. We note that the periodicities observed in the real-space spin textures are not attributed to incommensurate modulations within the Brillouin zone [106,107]. Instead, these commensurate periodicities are predetermined by the specific form of the exchange interaction $J\left(\mathbf{q}\right)$ in momentum space, which selects the ordering vector at $Q=\pi /3$. Therefore, the resulting spin textures naturally lock into simple commensurate periods without invoking additional lock-in effects or artifacts of the finite simulation box. This phase is topologically trivial and serves as a reference state for more complex multiple-Q phases.

Upon increasing K at small D, the system undergoes a transition from the single-Q spiral phase to the double-Q vortex crystal phase. In this state, two symmetry-related ordering wave vectors condense simultaneously, producing a periodic array of vortices and antivortices in the spin texture, as found by the real-space spin configuration in Figure 3b. The corresponding spin structure factor shown in Figure 2b exhibits two pairs of sharp Bragg peaks at $\pm {\mathit{Q}}_1$ and $\pm {\mathit{Q}}_2$, signaling the spontaneous breaking of the threefold rotational symmetry of the triangular-lattice system. Because the spin configuration remains coplanar, the scalar spin chirality vanishes, and the phase is topologically trivial despite its vortex–antivortex structure.

At larger values of K, two distinct triple-Q vortex crystal phases appear. They are classified into type I and type II, which can be clearly distinguished by their spin structure factors and real-space spin textures. In the triple-Q vortex crystal I phase, which emerges at relatively small D, the three ordering wave vectors contribute unequally, leading to anisotropic intensity distributions in the spin structure factor [Figure 2c]. This signals a spontaneous breaking of the threefold rotational symmetry, analogous to the double-Q vortex crystal. The corresponding real-space spin configuration consists of a vortex-antivortex lattice where the positions of vortex cores lack threefold rotational symmetry, giving rise to a distorted triangular arrangement, as shown in Figure 3c. This state can thus be regarded as an intermediate phase between the double-Q vortex crystal and the fully symmetric triple-Q configurations corresponding to the triple-Q vortex crystal II described below.

In contrast, the triple-Q vortex crystal II phase emerges at larger values of D, where the three ordering vectors ${\mathit{Q}}_1$, ${\mathit{Q}}_2$, and ${\mathit{Q}}_3$ contribute equally. In this case, the spin structure factor exhibits three Bragg peaks of identical intensity [Figure 2d], demonstrating the restoration of the threefold rotational symmetry of the triangular lattice. The real-space spin texture shown in Figure 3d reveals a highly ordered triangular array of vortices and antivortices, with the vortex cores arranged periodically to maintain the threefold rotational symmetry. Compared to the triple-Q vortex crystal I state, the triple-Q vortex crystal II state therefore exhibits a more symmetric and robust vortex–crystal arrangement, stabilized by the larger DM interaction, which enforces equal weight among the three symmetry-related ordering wave vectors. Notably, similar triple-Q vortex crystals preserving threefold rotational symmetry have been reported in frustrated spin models of centrosymmetric systems subjected to an external magnetic field [108,109].

The evolution of phases with increasing K at weak D can therefore be summarized as a sequence from the single-Q spiral state to the double-Q vortex crystal, followed by the triple-Q vortex crystal I and finally the triple-Q vortex crystal II, with the threefold rotational symmetry gradually restored as K increases. Conversely, strengthening the DM interaction at fixed K suppresses multiple-Q superpositions and favors the single-Q spiral state. This reflects the fact that the DM interaction stabilizes a unique single-Q spiral order by fixing the spiral rotation, whereas the biquadratic interaction promotes the coexistence of multiple ordering wave vectors, thereby driving the emergence of multiple-Q vortex crystals. The transitions between these magnetic phases are generally of first-order character, as expected from the absence of group–subgroup relations among the competing orders. In our simulated annealing calculations, the phase boundaries are identified from abrupt changes in order parameters such as magnetization, scalar spin chirality, and spin structure factors. While this approach allows us to delimit equilibrium phase boundaries, the precise determination of spinodal and binodal lines is beyond the present scope, since metastability effects and finite-size limitations prevent us from accessing such details in a strict thermodynamic sense.

Overall, the phase diagram highlights the delicate balance between the DM and biquadratic interactions in shaping the landscape of vortex-type spin textures. The identification of two distinct triple-Q vortex crystals demonstrates that even within coplanar spin configurations enforced by easy-plane anisotropy, symmetry-distinct vortex crystals can arise. These results suggest that out-of-plane DM interactions in $D_{3\mathrm{h}}$-symmetric systems provide a promising route to engineer unconventional multiple-Q states, offering theoretical guidance for exploring real materials where diverse vortex crystals and their characteristic responses may be realized.

3.2. Effect of Single-Ion Anisotropy

Next, we examine how the easy-plane single-ion anisotropy A affects the stability of vortex crystals, focusing in particular on the two types of triple-Q vortex crystals. For weak DM interaction ($D=0.04$) and moderate $K=0.6$, large negative A stabilizes the triple-Q vortex crystal I phase, as shown in Figure 4a. Its spin structure factor shows three Bragg intensities of unequal magnitude at ${\mathit{Q}}_1$–${\mathit{Q}}_3$, as shown in Figure 4b, indicating spontaneous breaking of the threefold rotational symmetry, which is consistent with the results in Figure 2c.

As A increases toward zero, the system undergoes a transition into a skyrmion crystal with the skyrmion number of $\pm 2$, which is indicated by the onset of finite scalar spin chirality ${\chi }^{\mathrm{sc}}$ in Figure 4a. The skyrmion number $n_{\mathrm{sk}}$ corresponding to the topological charge $Q_{\mathrm{sp}}$ is defined as

$$\begin{array}{c}\hfill n_{\mathrm{sk}}={\displaystyle \frac{1}{4\pi N_{\mathrm{m}}}}〈\sum _{\mathit{R}}{\Omega }_{\mathit{R}}〉,\end{array}$$

(11)

where $N_{\mathrm{m}}$ is the number of magnetic unit cells and ${\Omega }_{\mathit{R}}$ is the skyrmion density at plaquette $\mathit{R}$ [110]:

$$\begin{array}{c}\hfill tan\left({\displaystyle \frac{{\Omega }_{\mathit{R}}}{2}}\right)={\displaystyle \frac{{\mathit{S}}_i·({\mathit{S}}_j\times {\mathit{S}}_k)}{1+{\mathit{S}}_i·{\mathit{S}}_j+{\mathit{S}}_j·{\mathit{S}}_k+{\mathit{S}}_k·{\mathit{S}}_i}}.\end{array}$$

(12)

This equation indicates that the skrymion number $n_{\mathrm{sk}}$ is related to the scalar spin chirality in Equation (10). By summing the local scalar spin chirality over all plaquettes in the magnetic unit cell and normalizing by $4\pi$, we directly obtain the skyrmion number corresponding to the topological charge. As a result of the calculations, we refer to this state as the SkX-2 I. Although the behavior of squared magnetic moments ${\left(m_{{\mathit{Q}}_{\nu }}\right)}^2$ is continuously connected to the triple-Q vortex crystal I, the finite z-spin components generate noncoplanar spin textures with a nonzero skyrmion number, as shown in Figure 5a. The continuous variations of both $M^z$ and ${\chi }^{\mathrm{sc}}$ across the phase boundary suggest that this transition is of second-order character. We note, however, that in finite-size simulations, weakly first-order transitions may also appear continuous, and a more detailed finite-size scaling analysis would be required to rigorously confirm the transition order. This $n_{\mathrm{sk}}=2$ skyrmion crystal has analogs in square-lattice models with compass-type anisotropy and/or biquadratic interactions [111,112]. Because bond-dependent anisotropy is absent, states with positive and negative $n_{\mathrm{sk}}$ are energetically degenerate [113]. This state also exhibits a finite net $M^z$, as shown in Figure 4a. As A increases, the modulation of the z-spin component becomes more pronounced compared with the spin configurations shown in Figure 5a,b. Accordingly, the scalar spin chirality is also more pronounced compared with the configurations shown in Figure 6a,b. In the SkX-2 I phase, the squared magnetic moments at ${\mathit{Q}}_2$ and ${\mathit{Q}}_3$ exhibit non-monotonic variations as a function of A. This behavior might originate from the continuous redistribution of spectral weight among the three ordering vectors during the crossover from the triple-Q vortex crystal I to the SkX-2 II state, the latter of which appears for smaller $\left|A\right|$. The local extremum of ${\mathit{Q}}_2$ (or ${\mathit{Q}}_3$) does not signify a phase transition but rather a smooth rearrangement of the real-space skyrmion spin configuration.

At weaker $\left|A\right|$, the system transitions to another skyrmion crystal, denoted SkX-2 II, characterized by threefold-symmetric Bragg peaks of equal intensity at ${\mathit{Q}}_1$–${\mathit{Q}}_3$, as shown in Figure 4b. The threefold symmetric Bragg peaks imply that the spin ansatz of this state is given by

$$\begin{array}{c}\hfill {\mathit{S}}_i\propto {\left(\begin{array}{c}{\displaystyle \frac{\sqrt{3}}{2}}(cos{\mathit{Q}}_2·{\mathit{r}}_i-cos{\mathit{Q}}_3·{\mathit{r}}_i)\\ cos{\mathit{Q}}_1·{\mathit{r}}_i-{\displaystyle \frac{1}{2}}(cos{\mathit{Q}}_2·{\mathit{r}}_i+cos{\mathit{Q}}_3·{\mathit{r}}_i)\\ {\displaystyle \frac{1}{\sqrt{2}}}(cos{\mathit{Q}}_1·{\mathit{r}}_i+cos{\mathit{Q}}_2·{\mathit{r}}_i+cos{\mathit{Q}}_3·{\mathit{r}}_i)\end{array}\right)}^{\mathrm{T}}.\end{array}$$

(13)

This expression indicates that the spin configuration of the SkX-2 II is represented by a superposition of triple-Q sinusoidal waves at ${\mathit{Q}}_1$–${\mathit{Q}}_3$, giving rise to the real-space texture shown in Figure 5c and scalar spin chirality texture shown in Figure 6c. Such a spin configuration has also been reported even in a centrosymmetric spin model incorporating the effect of the biquadratic interaction and easy-plane anisotropy but without the DM interaction [114].

As A approaches zero, the system undergoes another transition to the SkX-2 III phase, as shown in Figure 5d and Figure 6d. We refer to this state as the SkX-2 III. The magnitudes of ${\left(m_{{\mathit{Q}}_{\nu }}\right)}^2$ at ${\mathit{Q}}_1$–${\mathit{Q}}_3$ are slightly different for $A\ne 0$, but they become the same for $A=0$, as shown in Figure 4b. The spin ansatz of this state is given by

$$\begin{array}{c}\hfill {\mathit{S}}_i\propto {\left(\begin{array}{c}cos{\mathit{Q}}_1·{\mathit{r}}_i\\ cos{\mathit{Q}}_2·{\mathit{r}}_i\\ a_{z}cos{\mathit{Q}}_3·{\mathit{r}}_i\end{array}\right)}^{\mathrm{T}},\end{array}$$

(14)

where the parameter $a_z$ depends on A, and $a_z=1$ ($a_z<1$) for $A=0$ ($A<0$). Similar to Equation (13), this spin configuration is also described by a superposition of triple-Q sinusoidal waves at ${\mathit{Q}}_1$–${\mathit{Q}}_3$, but with a distinct pattern of spin density wave superposition. The continuous transition between SkX-2 II and SkX-2 III with varying A has also been observed in the Kondo lattice models with single-ion anisotropy [115].

For intermediate DM interaction at $D=0.2$ and $K=0.6$, the system at large negative A stabilizes the triple-Q vortex crystal II phase. As A is increased, the energy gain by A gradually decreases, and hence, the out-of-plane spin modulations and the associated scalar spin chirality appear, as shown by the A dependence of the magnetization and scalar spin chirality in Figure 7a and the square magnetic moments in Figure 7b. In other words, the system undergoes a phase transition into a noncoplanar state with nonzero scalar spin chirality, where the real-space spin configurations are presented in the presence of A in Figure 8a and in the absence of A in Figure 8b; the threefold rotational symmetry of the spin structure is broken. The observed continuous variations in $M^z$ and ${\chi }^{\mathrm{sc}}$ across the phase boundary imply that the transition is likely of second-order type. By calculating the skyrmion number of this state, one finds that the spin configuration exhibits the skyrmion number of $\pm 1$, which is in contrast to the skyrmion crystal with the skyrmion number of two obtained for small D. We refer to this state as the SkX-1. By looking into the real-space spin configuration, this skyrmion crystal is characterized by two types of meron with the same sign of the scalar spin chirality, that is, the bimeron structure [116,117]. It is noted that the present bimeron structure is stabilized even without the external magnetic field, which is distinct from the previous one [116,117]. In other words, the interplay between the biquadratic interaction and the out-of-plane DM interaction can lead to the zero-field skyrmion crystal with $n_{\mathrm{sk}}=\pm 1$ when the magnitude of the easy-plane single-ion anisotropy is weak [118].

For strong DM interaction ($D=0.5$) combined with large $K=0.8$, the system exhibits qualitatively different behavior. In this regime, the triple-Q vortex crystal II phase is robustly stabilized across the entire range of A, as shown in Figure 9. The Bragg intensities at ${\mathit{Q}}_1$–${\mathit{Q}}_3$ remain nearly unchanged with varying A, and both ${\chi }^{\mathrm{sc}}$ and $M^z$ stay vanishingly small (not shown). This robustness indicates that once the DM interaction and the biquadratic interaction become sufficiently strong, they enforce a coplanar triple-Q state with high symmetry, effectively suppressing any topological instability toward skyrmion formation. In other words, the large D and K parameters lock the system into a vortex–crystal configuration that is insensitive to the competing effects of single-ion anisotropy.

Taken together, these results clarify the dual role of A and D in controlling the stability of vortex and skyrmion crystals. Negative A generally favors coplanar vortex crystals by enforcing in-plane spin alignment, while reducing $\left|A\right|$ relaxes this constraint and allows the emergence of noncoplanar textures that stabilize skyrmion crystals. The skyrmion number realized depends sensitively on the DM interaction D: for weak DM ($D=0.04$), the system hosts the skyrmion crystals with $n_{\mathrm{sk}}=2$, for intermediate DM ($D=0.2$), the skyrmion crystals with $n_{\mathrm{sk}}=1$ emerges, and for strong DM ($D=0.5$), skyrmion crystals are completely suppressed as the triple-Q vortex crystal II phase dominates.

These systematic trends establish single-ion anisotropy A as an essential tuning parameter for controlling the phase transition between vortex crystals and skyrmion crystals. At the same time, the DM interaction D acts as a selector of the skyrmion number, thereby governing the topological character of the emergent spin textures. The cooperative and competitive interplay of A and D thus provides a framework for engineering a variety of nontrivial magnetic phases in noncentrosymmetric magnets with $D_{3\mathrm{h}}$ symmetry.

3.3. Effect of Magnetic Field

We next investigate the influence of an out-of-plane magnetic field H on the stability of vortex crystal phases. Figure 10 and Figure 11 summarize the low-temperature results against H at fixed $A=-0.4$, $D=0.2$, and $K=0.6$, where the triple-Q vortex crystal II state is stabilized at zero field. As shown in Figure 10a, the uniform magnetization $M^z$ increases monotonically with increasing H, exhibiting small kinks that signal successive phase transitions. By contrast, the corresponding scalar spin chirality ${\chi }^{\mathrm{sc}}$ remains nearly zero throughout the entire field range, indicating the absence of topologically nontrivial spin textures in this parameter regime.

The evolution of the spin structure factor in Figure 10b reveals a clear sequence of magnetic phases. At low fields, the system resides in the triple-Q vortex crystal II phase, characterized by three symmetry-related Bragg peaks of equal intensity. Upon increasing H, the system undergoes a transition to the single-Q spiral phase, in which only a single pair of Bragg peaks persists, as shown in Figure 2a. At sufficiently large H, these finite-Q peaks vanish, signaling the phase transition into the fully polarized state.

Real-space spin textures corresponding to these phases are displayed in Figure 11. At $H=1.5$, the spin configuration in Figure 11a exhibits a well-ordered triangular lattice of vortices and antivortices, consistent with the triple-Q vortex crystal II, as shown in Figure 3d. At $H=1.7$, the spin pattern evolves into a spiral texture with a dominant modulation along a single ordering wave vector [Figure 11b], as expected in the single-Q spiral phase shown in Figure 3a. At even higher fields, all spins are uniformly aligned with the magnetic field, resulting in the fully polarized state. These findings demonstrate that the out-of-plane magnetic field destabilizes the multiple-Q vortex crystal states, driving the system through a continuous sequence from the triple-Q vortex crystal II to the single-Q spiral state, and finally into the fully polarized state.

4. Conclusions

We have shown that a momentum-space effective spin model on a triangular lattice with $D_{3\mathrm{h}}$ symmetry, comprising an out-of-plane DM interaction $\left(D\right)$, a positive biquadratic coupling $\left(K\right)$, and easy-plane single-ion anisotropy $\left(A\right)$, stabilizes a rich hierarchy of vortex-type multiple-Q states at low temperatures. In the $(D,K)$ plane, four ordered phases are realized: the single-Q spiral state, the double-Q vortex crystal, and two symmetry-inequivalent triple-Q vortex crystals. Among them, one of the triple-Q vortex crystals emerges only when both D and K are finite, underscoring the cooperative role of these two interactions.

Tuning the easy-plane single-ion anisotropy A controls the phase transition between coplanar vortex crystals and noncoplanar skyrmion crystals. For weak DM interaction, weakening the magnitude of the single-ion anisotropy converts the anisotropic triple-Q vortex crystal into skyrmion crystals with the skyrmion number of two, while for intermediate DM interaction, an analogous route yields the skyrmion crystal with the skyrmion number of one from the isotropic triple-Q vortex crystal. By contrast, for strong DM interaction, the threefold-symmetric triple-Q vortex crystal remains robust against variations in A, indicating that large D enforces coplanar order and suppresses topological transitions. These trends show that the DM interaction, as well as the easy-axis single-ion anisotropy, primarily determine the degree of out-of-plane spin modulation and scalar spin chirality, thereby enabling the stabilization of skyrmion textures. Applying an external out-of-plane magnetic field H further drives the system through a sequence of phases: from the triple-Q vortex crystal at low fields, to the single-Q spiral state at intermediate fields, and finally to the fully polarized state at high fields. In this process, the uniform magnetization increases monotonically with H, while the scalar spin chirality remains negligible, reflecting the suppression of nontrivial topology under strong field alignment.

Altogether, our findings identify the out-of-plane DM interaction allowed by $D_{3\mathrm{h}}$ symmetry as a key ingredient for realizing unconventional vortex crystal states that are inaccessible in purely polar or chiral magnetic systems. This establishes a new design principle for engineering exotic magnetic textures, highlighting noncentrosymmetric magnets with $D_{3\mathrm{h}}$ symmetry as promising platforms for exploring emergent electrodynamics and topological responses.