Modal Mode Simulation of Near-Unstable Cavities with Realistic Mirror Maps

Abstract

Near-unstable cavities hold promise for reducing thermal noise in next-generation gravitational wave detectors and for enhancing light–matter interactions in quantum electrodynamics. However, operating close to the edge of geometrical stability presents significant challenges, including increased coupling to higher-order modes and heightened sensitivity to small cavity length changes and mirror imperfections. This study employs Finesse v3 simulations to systematically investigate the modal behavior of a plano-concave cavity as it approaches instability, incorporating measured mirror surface defects and anisotropic curvature to replicate realistic conditions. The simulations highlight the degradation of beam purity and control signals as the cavity approaches instability. By validating the simulations against experimental data, we confirm Finesse’s reliability for modeling cavities while identifying critical limitations in regimes close to the edge of stability. These findings provide essential guidance for optimizing cavity designs in future gravitational wave detectors, balancing performance gains against the challenges of operating at the stability edge.

1. Introduction

Gravitational wave detection has become a cornerstone for testing general relativity and studying compact astrophysical objects [1,2,3]. As one of the major noise sources in advanced gravitational wave detectors, thermal noise significantly impacts detector sensitivity across most target frequency bands [4,5,6]. To meet the ever-increasing sensitivity requirements of next-generation detectors, suppressing thermal noise, especially coating thermal noise, has emerged as a critical research focus [7,8].

Optical resonators operating near the stability boundary (known as near-unstable cavities, NUCs) have emerged as promising candidates for next-generation gravitational wave detection systems, owing to their advantageous combination of compact geometry and expanded beam profiles that effectively mitigate thermal noise limitations [9,10]. Beyond their application in precision interferometry, such marginally stable optical cavities have become indispensable tools in quantum optical experiments, particularly for achieving strong light–matter coupling in cavity quantum electrodynamics and atomic physics implementations [11,12,13,14,15,16]. The enhanced sensitivity of these configurations comes with inherent challenges, as their proximity to the stability threshold renders them vulnerable to even minute variations in either the resonator length or mirror surface characteristics. They also face challenges such as high optical losses and degeneracy of Gaussian modes. Additionally, the well-defined Gaussian beams can become distorted through interactions with small imperfections on mirror surfaces. These factors negatively affect the instrument’s sensitivity and controllability.

1.1. Stability Criteria and near Unstable Cavity

The stability of an optical resonator determines whether a laser beam can maintain stable propagation through multiple reflections. In simple terms, if the beam remains confined within finite dimensions after successive reflections, the cavity is considered “stable”. Conversely, if the beam diverges and eventually escapes the cavity, it is classified as “unstable”. The stability condition is independent of mirror reflectivity but is intrinsically linked to the curvature radii of the mirrors and the cavity length. Take the simple linear two-mirror cavity as an example. Suppose that the curvature radii of the front mirror and the end mirror are $R_1$ and $R_2$, respectively, and the cavity length is L. The stability of the cavity can be characterized by the g-factor, which is defined as

$$g=\left(1-{\displaystyle \frac{L}{R_1}}\right)\left(1-{\displaystyle \frac{L}{R_2}}\right).$$

(1)

A cavity is stable when $0\le g\le 1$. Violation of this condition results in an unstable configuration in which the beam cannot be sustained. In particular, stable cavities exhibit well-defined eigenmodes and high cavity gain, making them ideal for interferometric applications. Unstable cavities can be beneficial in specific applications, such as high-power lasers, where their large mode volumes can result in higher gain [17,18]. However, analyzing mode behavior in these cavities is particularly challenging. Various techniques for characterizing such modes are described in [19,20]. NUCs represent a special class of optical resonators that operate at the boundary of the stability condition where g approaches 0 or 1. These configurations exhibit special beam properties that make them valuable for precision interferometry in gravitational wave detection.

For the concentric cavities widely used in current gravitational wave detectors, the mirror curvature radii satisfy $R_1=R_2=R_{\mathrm{c}}$ with a cavity length of $L=2R_{\mathrm{c}}$. The beam size on the mirrors can be expressed as

$$w=\sqrt{{\displaystyle \frac{\lambda R_c}{\pi }}}\sqrt{{\displaystyle \frac{1+\sqrt{g}}{1-\sqrt{g}}}}.$$

(2)

This reveals that when the cavity approaches the stability boundary ($g\sim 1$), the beam spot expands dramatically. This behavior stems from the fundamental dependence of beam parameters on the g-factor, where the spot size grows significantly as the cavity nears instability. The expanded beam size provides a direct mechanism for reducing coating thermal noise, since the noise power spectral density decreases with the square of the beam spot size. However, these benefits come with important technical considerations.

1.2. Challenges of Using near Unstable Cavity

The same geometric properties that enable large beam sizes make the cavities extremely sensitive to tiny perturbations. Cavity length changes on the order of micrometers or subtle mirror imperfections that would be negligible in stable cavities can destabilize a near unstable configuration. Marginally stable optical resonators exhibit enhanced fundamental-to-higher-order mode coupling from mirror surface defects (such as figure errors, surface roughness, and coating non-uniformity) compared to their stable counterparts. This coupling often results in energy being diverted from the fundamental mode to higher-order modes, effectively causing a loss and reducing the overall cavity gain.

Additionally, Gaussian mode degeneracy arises when there is insufficient Gouy phase separation between modes. This modal degeneracy significantly influences both spectral characteristics and energy distribution, driving inevitable power exchange among co-resonant states (see Section 3 for detailed analysis). While conventional stable configurations effectively filter out higher-order modes, marginally stable resonators permit their sustained oscillation due to mirror-induced perturbations. The cascading interaction between successively excited modes progressively degrades beam quality, while the resulting multimodal interference introduces systematic errors in length stabilization and alignment feedback systems.

These fundamental trade-offs present the key engineering challenge for NUCs in gravitational wave detectors. On one hand, they offer better thermal noise performance. On the other hand, they become more sensitive to perturbations. Our research aims to precisely quantify this balance through detailed simulations. In next-generation detectors, coating thermal noise is expected to be a major limiting factor. Despite their operational complexities, NUCs could provide crucial benefits for these future detectors.

In this work, we employ Finesse simulations to systematically investigate modal behavior in near-unstable plano-concave cavities, incorporating measured mirror surface defects and anisotropic curvature to replicate experimental conditions. Our study progresses from establishing the theoretical framework of cavity stability to analyzing higher-order mode coupling effects through comprehensive frequency scans, ultimately validating the simulations against experimental data. The results quantitatively characterize the stability as the cavity approaches geometric instability, providing critical insights for future gravitational wave detector designs.

2. Model of the Simulation

2.1. Finesse Simulation Framework with Higher-Order Modes

Finesse [21] is a versatile interferometer simulation tool that enables modeling of complex optical systems through numerical solution of steady-state field amplitudes. It supports both plane-wave and Hermite–Gaussian mode analyses, with the latter enabling precise studies of mode-matching and misalignment effects. The program also allows the construction of virtual interferometers using customizable optical components (lasers, mirrors, modulators, and detectors). It generates comprehensive output data for analysis, including detector signals, cavity eigenmodes, and beam parameters as functions of adjustable system parameters.

For modeling mirror surface imperfections and their effects, Finesse employs a modal decomposition approach. Any complex optical field $u(x,y,z)$ can be expressed as a superposition of orthogonal basis modes:

$$u(x,y,z)=\sum _{n,m}{k}_{nm}{u}_{nm}(x,y,z),$$

(3)

where $k_{nm}$ represents the coupling coefficients containing amplitude and phase information for each Hermite–Gauss (HG) or Laguerre–Gauss (LG) mode $u_{nm}$. The coupling coefficients are determined through orthogonality conditions

$$k_{nm}=\int \int u(x,y,z)u_{nm}^{\ast }(x,y,z)\mathrm{d}x\mathrm{d}y.$$

(4)

Mirror surface distortions primarily affect the beam through phase modulation, analogous to how water ripples distort reflections. This effect is quantified through generalized coupling coefficients that describe energy transfer between incident mode $u_{n^{\prime }{m}^{\prime }}$ and reflected/transmitted mode $u_{nm}$

$$k_{n,m,n^{\prime },m^{\prime }}\left(q,q^{\prime },A\right)=\int \int u_{nm}(x,y;q)A(x,y)u_{n^{\prime }{m}^{\prime }}^{\ast }\left(x,y;q^{\prime }\right)\mathrm{d}x\mathrm{d}y.$$

(5)

Here, q and $q^{\prime }$ represent complex beam parameters, while $A(x,y)$ encodes the mirror distortion function. For reflections, $A(x,y)=exp\left(2ik{n}_{1}z(x,y)\right)$, with $n_1$ being the refractive index and $z(x,y)$ the surface height profile. This formulation captures how mirror imperfections scatter light into higher-order modes, with the coupling coefficients quantifying the efficiency of this process.

Finesse implements this theoretical framework through numerical integration methods, allowing researchers to study how realistic mirror maps affect beam quality in gravitational wave detectors. This simulation program has been extensively validated through its use in Advanced LIGO [5,22], Advanced Virgo [6,23], KAGRA [24,25], and Einstein Telescope [26], establishing it as a reliable tool for investigating NUC behavior and optimizing future detector designs.

2.2. A Plano-Concave Cavity Model

To examine the behavior of cavity when it approaches the stability boundary and evaluate Finesse’s simulation capability for NUCs, we built a linear plano-concave cavity model using Finesse. By gradually adjusting the cavity length toward the unstable region and scanning across a specified laser frequency range, we observed changes in both fundamental and higher-order modes. During simulations, we not only tuned the cavity length and laser frequency but also investigated how increasing the maximum mode order ($maxtem$, the optional maximum mode order applied in Finesse simulation), affected the results. In order to facilitate comparison with the NUCs’ experimental results in Wang et al. [27], our simulation model adopts the same parameters as those in Wang et al. [27], shown in

Table 1. Main parameters of plano-concave cavity model.

Parameters	Values
Laser Power	1 W
Modulated Frequency	1 MHz
Modulation Type	Phase Modulation
Cavity Length	∼1 m
Radius of Curvature of IM	Flat
Radius of Curvature of EM $R_{cx}$	1.0012849 m
Radius of Curvature of EM $R_{cy}$	1.00114 m
Reflectivity of IM	0.9985
Reflectivity of EM	0.9985
FSR	∼150.05 MHz
Finesse	2092.82

.

The plano-concave cavity configuration replicates the experimental setup in Wang et al. [27], consisting of a flat input mirror (IM) and a concave end mirror (EM) with identical specifications. The initial cavity length, mirror curvatures, and reflectivities precisely match the experimental parameters from Wang et al. [27]. The mirror surface map shown in Figure 1 was directly measured from the experimental setup.

Both the plano-concave cavity and the widely used concentric cavity can be operated near the stability boundary while maintaining identical eigenmode properties. The key difference lies in their physical dimensions. The plano-concave configuration requires only half the cavity length of its concentric counterpart, consequently accumulating half the round-trip Gouy phase [27,28], which could be written as

$$\mathsf{\Delta }{\mathsf{\Psi }}_N=N·2\arctan \left({\displaystyle \frac{L}{z_R}}\right),$$

(6)

where N is the order of the mode, and the Rayleigh range $z_R$ for this geometry is $z_R=L\sqrt{g_2/\left(1-g_2\right)}$. The offset of the second-order mode resonance frequency from the fundamental mode is represented by $f_2$, which serves as a critical stability metric, follows from these relations as

$$f_2=2{\displaystyle \frac{\mathrm{FSR}}{\pi }}\arctan {\displaystyle \frac{L}{z_R}},$$

(7)

where FSR stands for the free spectral range, enabling derivation of the stability parameter and cavity length

$$L=R_2\left(1-g_2\right)={\displaystyle \frac{R_2}{2}}\left[1-\cos \left({\displaystyle \frac{f_2}{\mathrm{FSR}}}\pi \right)\right].$$

(8)

Remarkably, despite these differences, both cavity types exhibit identical eigenmode characteristics when operating near instability.

To accurately reproduce the effects of realistic mirror defects on modal characteristics, we applied the measured mirror map data from Wang et al. [27] on the EM of our NUCs simulation. Figure 1 depicts the mirror surface deformation (mirror map) of the EM, where the x and y axes represent two orthogonal spatial dimensions relative to the mirror center. And the optical path depth represents the height of surface variations along the normal direction of the mirror surface. Furthermore, to better approximate the actual experimental conditions of non-ideal spherical mirror surface, we introduced a controlled curvature anisotropy of $\mathsf{\Delta }{R}_c\sim 145\mathsf{\mu }\mathrm{m}$ between the x and y axes of the EM.

3. Simulation Results and Discussion

After building the plano-concave cavity model in Finesse, we performed a frequency sweep across 140–151 MHz to characterize the resonant behavior of Gaussian modes at the initial cavity length. The resulting transmission beam profile at the EM output is presented in Figure 2, demonstrating good agreement with experimental observations.

The simulation results presented in Figure 2 demonstrate excellent agreement with the experimentally observed higher-order mode resonance splitting. Three fourth-order modes (${\mathrm{HG}}_{40},{\mathrm{HG}}_{22}$, and ${\mathrm{HG}}_{04}$) exhibit less obvious resonance peaks near 141 MHz, with characteristic frequency separations of approximately 0.3 MHz. Similarly, the second-order modes (${\mathrm{HG}}_{20}$ and ${\mathrm{HG}}_{02}$) split into two dominant peaks near 145.5 MHz with comparable 0.3 MHz spacing. Notably, a weak secondary resonance (marked by red dot 5 in Figure 2) appears between the second-order mode peaks, displaying slightly elongated beam profiles that suggest possible hybrid mode coupling.

Under ideal conditions, modes of the same order should exhibit degenerate resonances since they accumulate identical Gouy phase delays. The observed frequency splitting strongly indicates the presence of mirror surface imperfections. NUCs are particularly sensitive to such defects, even micrometer-scale height variations across the mirror surface can induce differential Gouy phase shifts during multiple round trips, ultimately leading to the measured mode separations. This effect becomes especially pronounced when operating close to the stability boundary, where small phase differences accumulate significantly over many cavity circulations.

To verify whether mirror surface imperfections caused the observed mode splitting, we performed a controlled simulation by removing the measured mirror map from the EM and equalizing the curvature radii in all directions. As shown in Figure 3, this modification completely eliminated the mode separation phenomenon, all fourth-order modes merged into a single peak at 141 MHz, while the second-order modes coalesced at 145.5 MHz. These results confirm that mirror defects constitute the primary cause of higher-order mode splitting, as orthogonal modes in an ideal spherical cavity maintain identical phase accumulation due to perfect symmetry.

The beam profiles of both second- and fourth-order modes degenerated into rotationally symmetric Laguerre–Gauss patterns after removing surface imperfections. This transformation demonstrates the destructive effect of mirror defects on the symmetry of the mode. It also indicates that under ideal conditions, NUCs are possible to suppress higher-order mode excitation through proper mode matching techniques. The comparative results between Figure 2 and Figure 3 provide direct evidence that mirror surface imperfections, rather than intrinsic cavity properties, drive the observed higher-order mode resonances splitting in near-unstable configurations.

In order to investigate the behavior of our NUC model as it approached the stability boundary, we progressively increase the cavity length from $L=0.999$ m to $L=1.001139$ m (The maximum length that Finesse can simulate before the cavity becomes unstable). The top figure in Figure 4 presents the simulated evolution of resonance frequencies and amplitudes for fundamental and higher-order modes as the cavity’s g-factor approached zero, corresponding to significantly reduced geometric stability. The resonant peak near the right side represents the ${\mathrm{HG}}_{00}$ mode, while the two relatively distinct peaks on the left correspond to the ${\mathrm{HG}}_{20}$ and ${\mathrm{HG}}_{02}$ modes, respectively. As the cavity length increases, more tiny formant peaks appear on the left side of the graph line, indicating that more high-order modes are gradually moving towards the fundamental mode. The fundamental mode demonstrates a gradual leftward frequency shift with increasing cavity length, though this shift becomes markedly nonlinear near the instability boundary. This nonlinear behavior might be due to the coupling of higher-order modes at critical stability. At this point, the fundamental mode is no longer a pure fundamental mode, and its changes no longer follow the previous trend.

As the cavity length increases, the frequency splitting between the ${\mathrm{HG}}_{20}$ and ${\mathrm{HG}}_{02}$ modes shows an increasing trend. In the region at $L=0.999$ m, their separation is approximately 0.28 MHz, while at $L=1.001036$ m, the separation expands to 0.35 MHz. This enhanced frequency splitting originates from the differential impact of mirror asymmetry on the phase accumulation of orthogonal modes. In the near-unstable region, the beam becomes more sensitive to mirror curvature variations, causing ${\mathrm{HG}}_{20}$ and ${\mathrm{HG}}_{02}$ to experience different Gouy phase delays, thereby widening their frequency separation.

In the bottom figure of Figure 4, we present the evolution of beam profiles at the end mirror across the sixteen cavity length configurations, with the left column showing the first eight configurations from $L=0.999$ m to $L=1.0005$ m ($L1$ to $L8$) and the right column displaying the remaining configurations up to $L=1.001139$ m ($L9$ to $L16$). The images reveal a progressive beam size expansion that accelerates markedly as the cavity approaches instability, particularly evident in the final two configurations where a minimal 60 $\mathsf{\mu }\mathrm{m}$ cavity length increase (from $L=1.001081$ m to $L=1.001139$ m) produces a doubling of the spot size. For consistent visualization, all beam profiles ($L1$ to $L15$) maintain a uniform scale of [−200 $w_0$, 200 $w_0$], while $L16$ adopts an expanded [−400 $w_0$, 400 $w_0$] range to accommodate its significantly larger mode structure, as specified in Figure 4. This dramatic nonlinear growth directly correlates with the cavity’s approach to the stability boundary, quantitatively demonstrating the theoretical prediction of beam size divergence near instability.

In Figure 5a, we present the resonance simulation results under different cavity lengths and g-factor corresponding to each cavity. We observed the mode bunching phenomenon in the critical stable region in this figure. The higher-order modes progressively converge in frequency with the fundamental mode as cavity approaches geometric instability, which leads to a decrease in the purity of the fundamental mode. Figure 5b gives a comprehensive comparison between experimental measurements and Finesse simulations of resonance frequency evolution as the cavity approaches instability. The measured data from Wang et al. [27] (shown as discrete points) and theoretical fits (colored lines) for both fundamental and second-order modes demonstrate excellent agreement in the stable regime (cavity lengths below 1.0005 m). The resonant frequency of the fundamental mode, shown in Figure 5a, exhibits nonlinear behavior as the cavity approaches instability. However, experimental measurements display a linear trend. This discrepancy arises because the Pound–Drever–Hall technique, used for cavity locking, introduces a shift in the error signal due to the overlap between higher-order modes and the fundamental mode. To enable a meaningful comparison between the simulation and the measured data, the modeled resonance frequencies of the fundamental mode in Figure 5b are calibrated by subtracting the frequency spacing between two adjacent fundamental modes (one near 150 MHz and the other near 0 Hz). After this calibration, the simulated resonance behavior of the fundamental mode becomes linear, aligning with the measurement results. The resonant frequencies of the second-order modes are also calibrated by subtracting their values from the resonant frequency of the fundamental mode near 0 Hz.

As shown in Figure 5b, there are noticeable differences between the simulated and measured resonant frequencies of the second-order modes. This discrepancy arises because the mirror map used in the simulation was obtained prior to coating. In practice, the mirror curvature can change after coating and installation into the mirror mount. From Equation (8), the resonant frequency of a second-order mode is given by

$$f_2={\displaystyle \frac{\mathrm{FSR}}{\pi }}·\arccos \left(1-{\displaystyle \frac{2L}{R_2}}\right).$$

(9)

Assuming a small deviation in mirror curvature, where $L\to R_2\approx 1$ m, $\delta R<{10}^{-3}$ m $\ll R_2$ and $R_2-L\gg \delta R^2$, the expression becomes

$$f_2={\displaystyle \frac{\mathrm{FSR}}{\pi }}·arccos\left(1-{\displaystyle \frac{2L}{R_2+\delta R}}\right)\approx {\displaystyle \frac{\mathrm{FSR}}{\pi }}·\left(1-2\sqrt{{\displaystyle \frac{R_2-L}{R_2}}}-\sqrt{{\displaystyle \frac{\delta R^2}{R_2(R_2-L)}}}\right).$$

(10)

This shows that a small change in mirror curvature leads to a measurable shift in the resonant frequencies. In a plano-concave cavity, such offsets can be effectively corrected by applying appropriate frequency shifts to the resonances of each mode. Figure 6 presents the comprehensive comparison between these calibrated simulations, experimental measurements, and theoretical predictions. The simulated resonant frequencies are adjusted by applying a fixed offset to align them with the measured resonance frequencies in the stable region. To enable more intuitive comparison between simulation and experimental results, in Figure 6 we performed simulations at precisely matched cavity lengths corresponding to each experimental configuration, with the maximum simulated cavity length reaching $L=1.00112$ m. The plot demonstrates excellent three-way agreement in stable operating regimes (L < 1.0005 m), where the simulated ${\mathrm{HG}}_{20}$ and ${\mathrm{HG}}_{02}$ modes (white markers) precisely track both experimental data (colored dots) and fitting (colored lines) for all three observables. As the cavity approaches instability (1.0008 m < L < 1.00112 m), the experimental resonances of second-order modes begin deviating from fitting predictions due to higher-order mode coupling effects, while the simulations maintain alignment with the idealized fitting curves.

In the stable region, the behavior of second-order modes is primarily influenced by astigmatism in the mirror curvature. The Finesse simulation results demonstrate excellent agreement with both theoretical predictions and experimental measurements in stable operating regimes, particularly for the ${\mathrm{HG}}_{00}$ mode resonance frequencies that closely follow the expected free spectral range (FSR) fitting results. This correspondence validates the simulation’s reliability for conventional cavity configurations. However, in the near-unstable region where mode bunching occurs, numerous higher-order modes, coupled from the fundamental mode due to local mirror defects, become resonant simultaneously. This overlapping resonance significantly complicates the characterization of individual mode behaviors even with realistic mirror maps.

These systematic comparisons with experimental data from Wang et al. [27] serve two crucial purposes: first, they validate our simulation approach in stable operating regimes, where the simulated ${\mathrm{HG}}_{20}$ and ${\mathrm{HG}}_{02}$ modes show excellent agreement with measurements. Second, the observed discrepancies near the stability boundary highlight phenomena requiring additional experimental investigation, particularly the effects of post-coating curvature changes noted in Wang et al. [27]. This experimental-simulation synergy provides a robust framework for evaluating near-unstable cavity behavior while identifying areas where modeling assumptions may require refinement.

Figure 7 presents the Pound–Drever–Hall error signals simulated by Finesse under different cavity length configurations. Several key evolutionary characteristics are observed as the cavity length increases from the stable regime (L = 0.999 m) towards the simulation boundary (L = 1.001139 m). First, the slope of the fundamental mode error signal (at around 0 MHz) remains clearly distinguishable, demonstrating that near-unstable cavities can still provide feasible feedback control conditions before reaching the instability threshold. Second, the error signals of second-order modes (${\mathrm{HG}}_{20}$ and ${\mathrm{HG}}_{02}$) and their 1 MHz modulation sidebands gradually converge toward the fundamental mode as the cavity length increases, showing that overlap begins to emerge when L > 1.00113 m with approximately 0.1 MHz offset introduced to the fundamental mode error signal, confirming that higher-order mode coupling contaminates the frequency-locking reference signal. This observed offset provides a direct explanation for the nonlinear frequency shift of the fundamental mode observed in Figure 5a as the cavity approaches the stability boundary, where progressive contamination from higher-order modes alters the expected resonance behavior. Finally, when the cavity length reaches 1.001139 m, the amplitude of the fundamental mode error signal exhibits over 90% attenuation (from $1.7\times {10}^{-7}$ [au]/Hz to $1.6\times {10}^{-8}$ [au]/Hz). Although the linear region of the error signal remains clearly distinguishable at this operating point, the substantial amplitude reduction significantly increases the challenges in control system implementation, particularly in maintaining the required locking stability for precision interferometry.

To investigate the influence of the maximum transverse mode order ($maxtem$) on NUCs simulations in Finesse, we systematically compare cavity scan results across different $maxtem$ settings. Figure 8 presents the comparison of resonance frequencies and amplitudes in a plano-concave cavity simulated with varying $maxtem$ as the cavity approaches the stability boundary. In the stable regime (Figure 8a, L = 0.999 m), increasing $maxtem$ from 10 to 16 has a negligible impact on the resonance spectrum, confirming that higher-order modes remain effectively decoupled from the fundamental mode when the cavity operates far from instability. However, as the cavity length increases toward the stability boundary (Figure 8b–d), the influence of $maxtem$ becomes pronounced. For L = 1.00112 m (g = 0.00016), additional resonance peaks emerge with higher $maxtem$, revealing previously unresolved higher-order modes. This trend intensifies in the near-unstable region ($L\ge 1.00112$ m, $g\le 0.00016$), where elevating $maxtem$ from 10 to 30 not only unveils a dense spectrum of higher-order resonances but also demonstrates a progressive reduction in the fundamental mode’s amplitude. This amplitude suppression reflects enhanced energy transfer from the fundamental mode to higher-order modes due to their degeneracy and strong coupling near instability. The results underscore the necessity of including sufficiently high $maxtem$ values in Finesse simulations to accurately capture the complex modal interactions in near-unstable cavities, particularly for predicting beam quality and control signal stability in gravitational wave detectors.

4. Conclusions

This study demonstrates the critical role of higher-order mode interactions in NUCs through systematic Finesse simulations incorporating realistic mirror imperfections. Our results reveal that while stable cavities exhibit negligible sensitivity to the $maxtem$, NUCs approaching the stability boundary show pronounced dependence on $maxtem$, with higher values unveiling dense spectra of coupled higher-order modes and significant energy transfer from the fundamental mode. The observed amplitude suppression of the fundamental mode and the emergence of additional resonances underscore the challenges of maintaining beam purity and control signal stability in NUCs. Furthermore, the excellent agreement between simulations and experimental data in stable regimes validates Finesse’s reliability, while discrepancies near instability highlight the need for comprehensive modeling of higher-order modes and mirror defects. While this work focused on plano-concave cavities to enable direct experimental validation, future studies should extend this framework to concentric and folded cavities. In particular, concentric cavities—used in current gravitational wave detectors—exhibit stronger higher-order mode degeneracy, where both odd- and even-order modes couple simultaneously to the fundamental mode. This introduces additional control challenges not present in plano-concave systems, requiring advanced techniques for mode suppression and alignment stabilization. These findings emphasize the trade-offs between thermal noise suppression and operational complexity in NUCs, providing essential insights for optimizing future gravitational wave detector designs. The study underscores the necessity of advanced simulation techniques to mitigate performance degradation in near-unstable configurations, ensuring their viability for next-generation precision interferometry. Future work will also focus on developing quantitative mode identification criteria to better characterize complex coupling effects in near-unstable configurations.