Symmetric Hybrid Hessian Adaptive Trust-Region Incremental Harmonic Balance Method for Strongly Nonlinear Systems

Abstract

To address the insufficient convergence robustness, strong dependence on the initial guess, and computational-efficiency bottlenecks of the incremental harmonic balance (IHB) method for strongly nonlinear systems, this paper develops an adaptive framework that integrates a trust-region strategy with a hybrid Hessian matrix. The framework reformulates the harmonic-balance iteration as a constrained nonlinear least-squares optimization problem, constructs a symmetric hybrid Hessian matrix by blending the Gauss–Newton and Newton directions, and uses a trust-region algorithm to adaptively regulate the step size, thereby jointly optimizing the iterative path and convergence behavior. Numerical results show that the proposed method significantly enhances convergence robustness, reduces sensitivity to initial guesses, and improves computational efficiency while maintaining high accuracy. It also captures both stable and unstable periodic solutions in strongly nonlinear systems.

1. Introduction

Vibration systems encountered in engineering practice often exhibit strong nonlinearities, and the reliable computation of their periodic responses and stability assessment is directly related to structural safety and performance design [1,2,3]. To address this challenge, purely analytical methods, such as the Lindstedt–Poincaré method [4,5], the method of multiple scales [6,7], and the averaging method [8,9], have a narrow range of applicability because they are restricted by weak-nonlinearity assumptions. Fully numerical methods, such as Runge–Kutta methods [10,11], the finite element method [12,13], and the boundary element method [14,15], face efficiency bottlenecks in studies involving parameter variations and multiple solution branches. Against this background, the incremental harmonic balance (IHB) method [16], a semi-analytical numerical approach that combines the advantages of analytical and numerical methods, has been developed. By integrating incremental linearization, Galerkin frequency-domain balancing, and Floquet stability theory, the IHB method enables efficient and high-accuracy computation of periodic responses in strongly nonlinear systems [17,18]. Its effectiveness has been demonstrated in several important fields, including mechanical engineering [19,20], civil engineering [21,22], and aerospace engineering [23,24].

Although the IHB method combines the strengths of analytical and numerical approaches, two major bottlenecks remain in practical applications. The first is limited computational efficiency: in each iteration, the error matrix and the Jacobian matrix must be reconstructed, leading to a heavy computational burden. To this end, a variety of acceleration strategies have been developed. Ni et al. [25] improved the computational efficiency of the Galerkin procedure using the fast Fourier transform; building on this, Ju et al. [26] further reduced the computational cost by introducing tensor contraction techniques; Li et al. [27] adopted quasi-Newton methods to adjust the search direction and thereby reduce the number of iterations. In addition, a system-decoupling strategy that applies harmonic balance only to the nonlinear components has been used to substantially reduce the computational scale of multi-degree-of-freedom problems [28]. The second bottleneck is that convergence depends strongly on the initial guess; initial values with large deviations often lead to iterative failure. To enhance convergence robustness, Cheung et al. [29] used an arc-length method to predict initial values and accelerate convergence; Chen and Liu [30] effectively expanded the convergence range by combining homotopy analysis with the IHB method. To address ill-conditioning arising during iterations, Zheng et al. [31] employed Tikhonov regularization. On this basis, Huang et al. [32] further applied the Levenberg–Marquardt algorithm to enhance the stability of the iterative process. In recent years, research in this area has shown two notable trends: (i) incorporating optimization strategies such as backtracking line search [27] and trust-region methods [33] to comprehensively improve convergence robustness; and (ii) exploiting higher-order tensor information [34] to enhance the efficiency and accuracy of iterative computations.

Despite these advances, trust-region and Levenberg–Marquardt-enhanced IHB methods still adopt a fixed Gauss–Newton-type search direction, focusing primarily on step-size control. This becomes a critical limitation in strongly nonlinear regimes, where the Gauss–Newton direction may be insufficient or even divergent. To address this, the present work proposes a symmetric hybrid Hessian matrix that adaptively blends the Gauss–Newton and Newton directions, enabling dynamic adjustment of the search direction based on the local landscape of the residual function. This search-direction optimization is integrated with FFT-based acceleration techniques to jointly improve computational efficiency and convergence robustness. Combined with a trust-region framework, this strategy achieves a coordinated optimization of both aspects, filling the gap left by prior methods that typically address either efficiency or stability in isolation.

The remainder of this paper is organized as follows. Section 2 reviews the conventional IHB method, including the incremental procedure, stability analysis, and arc-length continuation. Section 3 presents the proposed MHTR-IHB method, with emphasis on the construction of the symmetric hybrid Hessian matrix, the FFT-based acceleration scheme, and the trust-region update strategy. Section 4 provides three numerical examples to validate the accuracy, efficiency, and robustness of the proposed method. Finally, Section 5 summarizes the main conclusions.

2. Traditional Incremental Harmonic Balance Method

Consider a conventional multi-degree-of-freedom nonlinear system, whose governing equation of motion is:

$$\mathit{M}{\displaystyle \frac{d^2\mathit{q}}{d{t}^2}}+\mathit{C}{\displaystyle \frac{d\mathit{q}}{dt}}+\mathit{K}\mathit{q}+\mathit{G}(\mathit{q})=\mathit{F}\cos (m\omega t),$$

(1)

where $\mathit{q}={\left[q_1,q_2,\dots \dots q_n\right]}^{\mathrm{T}}$ is the displacement vector of the system; $\mathit{M},\mathit{C}$, and $\mathit{K}$ are the mass matrix, damping matrix, and linear stiffness matrix, respectively; $\mathit{G}\left(\mathit{q}\right)$ is the nonlinear force vector; $\mathit{F}$ is the excitation force vector; $m$ is the harmonic order of the excitation; and $\omega$ is the excitation frequency.

To directly solve for the response with period $\mathrm{T}$, introduce the time-scale transformation $\tau =\omega t$. Equation (1) can then be rewritten as:

$${\omega }^2\mathit{M}{\displaystyle \frac{d^2\mathit{q}}{d{\tau }^2}}+\omega \mathit{C}{\displaystyle \frac{d\mathit{q}}{d\tau }}+\mathit{K}\mathit{q}+\mathit{G}\left(\mathit{q}\right)=\mathit{F}\cos \left(m\tau \right).$$

(2)

2.1. Incremental Harmonic Balance Procedure

Let ${\mathit{q}}_0$ and ${\omega }_0$ denote an initial solution of Equation (2). In its neighborhood, the solution can be expressed in incremental form as:

$$\mathit{q}={\mathit{q}}_0+\Delta \mathit{q},\omega ={\omega }_0+\Delta \omega .$$

(3)

Linearizing Equation (1) at $\left(q_0,{\omega }_0\right)$ yields the incremental equation:

$${\omega }_0^2\mathit{M}{\displaystyle \frac{d^2\Delta \mathit{q}}{d{\tau }^2}}+{\omega }_0\mathit{C}{\displaystyle \frac{d\Delta \mathit{q}}{d\tau }}+(\mathit{K}+{\mathit{K}}_{\mathrm{nonlinear}})\Delta q+(2{\omega }_0{\displaystyle \frac{d^2{\mathit{q}}_0}{d{\tau }^2}}+C{\displaystyle \frac{d{\mathit{q}}_0}{d\tau }})\Delta \omega =-\mathit{f}\mathit{R}.$$

(4)

In the above equation, $\mathit{f}\mathit{R}$ denotes the unbalanced-force residual of the system, whereas ${\mathit{K}}_{\mathrm{nonlinear}}$ corresponds to the nonlinear stiffness matrix. Their explicit expressions can be written as:

$$\begin{array}{l}\mathit{f}\mathit{R}={\omega }_0^2\mathit{M}{\displaystyle \frac{d^2{\mathit{q}}_0}{d{\tau }^2}}+{\omega }_0\mathit{C}{\displaystyle \frac{d{\mathit{q}}_0}{d\tau }}+\mathit{K}{\mathit{q}}_0+\mathit{G}({\mathit{q}}_0)-\mathit{F}\cos (m\tau ),\\ {\mathit{K}}_{\mathrm{nonlinear}}={\left.{\displaystyle \frac{\partial \mathit{G}}{\partial \mathit{q}}}\right|}_{\mathit{q}={\mathit{q}}_0}.\end{array}$$

(5)

Within the framework of the incremental harmonic balance method, the periodic response is approximated by a harmonic expansion. Let $N_s$ be the truncation order. A basis-function matrix $\mathit{S}$ can be constructed as:

$$\mathit{S}=\mathrm{diag}\left({\mathit{C}}_S,{\mathit{C}}_S,\dots \dots ,{\mathit{C}}_S\right),$$

(6)

where

$$C_S=\left[1,\cos \tau ,\sin \tau ,\cos 2\tau ,\sin 2\tau ,\dots \dots ,\cos i\tau ,\sin i\tau ,\dots \dots ,\cos N_s\tau ,\sin N_s\tau \right].$$

Based on this expansion, the initial harmonic coefficient vector ${\mathit{A}}_0$ characterizing the system response and its incremental vector $\Delta \mathit{A}$ can be expressed as:

$${\mathit{q}}_0=\mathit{S}{\mathit{A}}_0,\Delta \mathit{q}=\mathit{S}\Delta \mathit{A},$$

(7)

where

$$\begin{array}{l}{\mathit{A}}_0={\left[a_{10},a_{11},b_{11},\dots \dots ,a_{1N},b_{1N},a_{20},\dots \dots ,a_{n{N}_s},b_{n{N}_{\mathrm{s}}}\right]}^{\mathrm{T}},\\ \Delta \mathit{A}={\left[\Delta a_{10},\Delta a_{11},\Delta b_{11},\dots \dots ,\Delta a_{1N},\Delta b_{1N},\Delta a_{20},\dots \dots ,\Delta a_{n{N}_s},\Delta b_{n{N}_{\mathrm{s}}}\right]}^{\mathrm{T}}.\end{array}$$

By applying the Galerkin projection, the incremental time-domain equation is transformed into the incremental frequency-domain equation:

$${\mathit{K}}_{\mathrm{mc}}\Delta \mathit{A}+{\mathit{R}}_{\mathrm{mc}}\Delta \omega +\mathit{R}=0,$$

(8)

where

$$\begin{array}{l}\mathit{R}={\displaystyle {\int }_0^{2\pi }{\mathit{S}}^{\mathrm{T}}\cdot \left({\omega }_0^2\mathit{M}\frac{{\mathrm{d}}^2{\mathit{q}}_0}{\mathrm{d}{\tau }^2}-\mathit{C}{\omega }_0\frac{\mathrm{d}{\mathit{q}}_0}{\mathrm{d}\tau }-\mathit{K}{\mathit{q}}_0-\mathit{G}\left({\mathit{q}}_0\right)-\mathit{F}\cos \left(m\tau \right)\right)}\mathrm{d}\tau ,\\ {\mathit{K}}_{\mathrm{mc}}={\displaystyle {\int }_0^{2\pi }{\mathit{S}}^{\mathrm{T}}\cdot \left({\omega }_0^2\mathit{M}\frac{{\mathrm{d}}^2\mathit{S}}{\mathrm{d}{\tau }^2}+\mathit{C}{\omega }_0\frac{\mathrm{d}\mathit{S}}{\mathrm{d}\tau }+(\mathit{K}+{\mathit{K}}_{\mathrm{nonlinear}})\mathit{S}\right)}\mathrm{d}\tau ,\\ {\mathit{R}}_{mc}={\displaystyle {\int }_0^{2\pi }{\mathit{S}}^{\mathrm{T}}}\cdot \left(2{\omega }_0\mathit{M}{\displaystyle \frac{{\mathrm{d}}^2{\mathit{q}}_0}{\mathrm{d}{\tau }^2}}+\mathit{C}{\displaystyle \frac{\mathrm{d}{\mathit{q}}_0}{\mathrm{d}\tau }}\right)d\tau .\end{array}$$

In the above linear matrix equation, $\mathit{R}$ is the error matrix, ${\mathit{K}}_{mc}$ is the Jacobian matrix, and ${\mathit{R}}_{mc}$ is the frequency matrix; all three are constant matrices within the current iteration step. If the external excitation frequency $\omega$ is known and fixed, then $\Delta \omega =0$, and the equation can be simplified as:

$${\mathit{K}}_{\mathrm{mc}}\Delta \mathit{A}+\mathit{R}=0.$$

(9)

The iterative point $\mathit{A}$ is updated via Equation (9) until the prescribed convergence criterion $\parallel \mathit{R}\parallel \le {10}^{-10}$ is satisfied.

2.2. Stability Assessment of the System

The stability of the system is assessed by introducing a small perturbation $\Delta \mathit{q}$ to the exact solution, whereby Equation (2) becomes:

$${\omega }_0^2\mathit{M}{\displaystyle \frac{{\mathrm{d}}^2\Delta \mathit{q}}{\mathrm{d}{\tau }^2}}+{\omega }_0\mathit{C}{\displaystyle \frac{\mathrm{d}\Delta \mathit{q}}{\mathrm{d}\tau }}+(\mathit{K}+{\mathit{K}}_{\mathrm{nonlinear}})\Delta \mathit{q}=0.$$

(10)

Equation (10) can be rewritten as:

$${\displaystyle \frac{\mathrm{d}\mathit{X}}{\mathrm{d}\tau }}=\mathit{Q}\left(\tau \right)\mathit{X},$$

(11)

where

$$\mathit{X}={\left[\begin{array}{cc}\Delta \mathit{q}& {\displaystyle \frac{\mathrm{d}\Delta \mathit{q}}{\mathrm{d}\tau }}\end{array}\right]}^{\mathrm{T}},\mathit{Q}=\left[\begin{array}{cc}\mathbf{0}& \mathbf{I}\\ -{\displaystyle \frac{{\mathit{M}}^{-1}\left(\mathit{K}+3{\mathit{K}}_{\mathrm{nonlinear}}\right)}{{\omega }^2}}& -{\displaystyle \frac{{\mathit{M}}^{-1}\mathit{C}}{\omega }}\end{array}\right].$$

Here, $\mathbf{I}$ is the identity matrix and $\mathbf{0}$ denotes a zero matrix. Each entry of $\mathit{Q}\left(2,1\right)$ is a periodic function with period $T=2\pi /\omega$. For Equation (11), there necessarily exists a matrix $\mathit{Z}$ composed of a set of fundamental solutions, which satisfies the following matrix differential equation:

$${\displaystyle \frac{\mathrm{d}\mathit{Z}}{\mathrm{d}\tau }}=\mathit{Q}\left(\tau \right)\mathit{Z}.$$

(12)

Since $\mathit{Q}\left(\tau \right)=\mathit{Q}\left(\tau +\mathrm{T}\right)$, $\mathit{Z}\left(\tau +\mathrm{T}\right)$ is also a fundamental matrix solution and can therefore be expressed as:

$$\mathit{Z}\left(\tau +\mathrm{T}\right)=\mathit{P}\mathit{Z}\left(\tau \right),$$

(13)

where $\mathit{P}$ is a nonsingular constant matrix. According to the Floquet theory [22], the stability of the periodic solution is determined by the eigenvalues of $\mathit{P}$: when the eigenvalues of $\mathit{P}$ lie within $\left(-1,1\right)$, the periodic solution of the system is stable; otherwise, the periodic solution is unstable. Divide the period T into $N_k$ subintervals. For the $k$-th subinterval, the time step is ${\Delta }_k={\tau }_k-{\tau }_{k-1}$. By computing the linearized state transition matrix over each subinterval and multiplying them in sequence, the matrix $\mathit{P}$ can be approximated as:

$$\mathit{P}=\mathit{Z}\left(\mathrm{T}\right)={\displaystyle \prod _{i=1}^{N_k}\left[\mathbf{I}+{\displaystyle \sum _{j=1}^{N_j}\frac{{\left({\displaystyle {\int }_{{\tau }_{\mathrm{k}-1}}^{{\tau }_k}\mathit{Q}\left(\xi \right)\mathrm{d}\xi }\right)}^j}{j!}}\right]}={\displaystyle \prod _{i=1}^{N_k}\left[\mathbf{I}+{\displaystyle \sum _{j=1}^{N_j}\frac{{\left({\Delta }_k{Q}_k\right)}^j}{j!}}\right]}.$$

(14)

In this paper, the stability analysis results for all periodic solutions are integrated into the response-curve plots, following the visual convention that solid lines represent stable periodic solutions, whereas dashed lines represent unstable periodic solutions.

2.3. Arc-Length Continuation Technique

In parametric frequency-response analysis, if the excitation frequency $\omega$ is directly used as the control parameter, tracking often fails near critical locations such as turning points or bifurcation points on the solution path, because the system Jacobian matrix ${\mathit{K}}_{\mathrm{mc}}$ tends to become singular. To ensure that a complete response curve can be obtained continuously and stably, the arc-length continuation algorithm is employed in this paper. This method extends the solution vector, which includes the displacement harmonic coefficients and the frequency, to $\mathit{x}={\left[\omega ,\mathit{A}\right]}^{\mathrm{T}}$. The distance between two adjacent solution points is the local arc length $S_i=‖{\mathit{x}}_i-{\mathit{x}}_{i-1}‖$, and the cumulative arc length is $t_i={\displaystyle \sum _{k=1}^i{S}_k}$. If the arc-length increment at the current point is $\Delta S_i$, then the predicted arc length for the next point is $t_{i+1}=t_i+\Delta S_i$. Using the four most recently obtained consecutive exact solutions ${\mathit{x}}_{i-3},{\mathit{x}}_{i-2},{\mathit{x}}_{i-1},{\mathit{x}}_i$ and their corresponding arc lengths, a cubic Lagrange interpolation is used to extrapolate the initial prediction for the next iteration:

$${\mathit{x}}_{i+1}={\displaystyle \sum _{r=l-3}^i\left({\displaystyle \prod _{\begin{array}{l}p=i-3\\ r\ne p\end{array}}^i\frac{t_{i+1}-t_p}{t_r-t_p}}\right)}{\mathit{x}}_r.$$

(15)

The adaptive control of the step size $\Delta S_i$ is crucial to the efficiency and stability of the algorithm. In this paper, the step size is dynamically adjusted according to the convergence behavior at the previous corrected point:

$$\Delta S_i=k{\displaystyle \frac{\Delta S_{i-1}}{n_1}},$$

(16)

where $k$ is the control-rate factor of the algorithm. In the IHB algorithm, $k$ is generally taken as 2; for MHTR-IHB, $k$ can be increased relative to IHB, typically within the range of 3 to 5. In this study, the arc-length step size is adaptively regulated based on the number of iterations $n_1$ required during the correction procedure. When the response curve is relatively smooth and convergence is fast, the algorithm automatically adopts a larger step size to improve computational efficiency. In regions with large curvature or strong nonlinearity, more iterations are required for convergence, and the step size is correspondingly reduced to maintain numerical stability. To further avoid the Jacobian matrix ${\mathit{K}}_{\mathrm{mc}}$ becoming singular near critical points, the algorithm dynamically determines the dominant iterative variable according to the offset direction of the predicted point, using the following criterion:

$$k_x=\underset{j}{\max }\left({\mathit{x}}_{i+1,j}-{\mathit{x}}_{i,j}\right).$$

(17)

When the frequency direction is identified as dominant (corresponding to $k_x=1$), the excitation frequency $\omega$ is kept unchanged in the current iteration step, and only the harmonic coefficient vector $\mathit{A}$ is iteratively updated. When a certain displacement component is identified as dominant (corresponding to $k_x>1$), the corresponding component in $\mathit{A}$ is fixed, and $\omega$ is instead treated as the unknown variable to be solved iteratively. To accommodate this, the system Jacobian matrix ${\mathit{K}}_{\mathrm{mc}}$, the displacement increment vector $\Delta \mathit{A}$, and the frequency increment $\Delta \omega$ must be adjusted accordingly, i.e.,

$$\begin{array}{l}{\mathit{K}}_{\mathrm{mc}}=\left[\begin{array}{ccc}{\mathit{K}}_{\mathrm{mc}}\left(:,1:k_x-1\right)& {\mathit{R}}_{\mathrm{mc}}& {\mathit{K}}_{\mathrm{mc}}\left(:,k_x+1:\mathrm{end}\right)\end{array}\right],\\ \Delta \mathit{A}=\left[\begin{array}{ccc}\Delta \mathit{A}\left(:,1:k_x-1\right)& 0& \Delta \mathit{A}\left(:,k_x+1:\mathrm{end}\right)\end{array}\right],\\ \Delta \omega =\Delta \mathit{A}\left(:,k_x\right).\end{array}$$

(18)

3. MHTR-IHB Method for Computing Periodic Responses

MHTR-IHB reformulates the iterative solution process of the IHB method as the optimization process of a nonlinear least-squares problem. It improves computational speed by incorporating the FFT algorithm and controls the iterative step size via a trust-region algorithm and an adaptive Hessian matrix, thereby significantly enhancing convergence.

3.1. Introduction of the Newton Search Direction

When $‖\mathit{R}‖=0$, a solution of the IHB method exists. Therefore, the original problem can be transformed into a nonlinear least-squares optimization problem:

$$acg\min f\left(\mathit{A}\right)={\displaystyle \frac{1}{2}}{‖\mathit{R}‖}^2.$$

(19)

The Newton search direction of Equation (19) is associated with the second-order Taylor term of the objective. Expanding Equation (19) in a Taylor series, its second-order Taylor term is:

$$\begin{array}{l}\mathit{H}\left(\Delta \mathit{A}\right)=\Delta {\mathit{A}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}\Delta \mathit{A}+2{\mathit{R}}^{\mathrm{T}}\left({\mathbf{I}}_{\left(2N_S+1\right)\times n}\otimes \Delta {\mathit{A}}^{\mathrm{T}}\right){\mathit{h}}_{\mathrm{mc}}\Delta \mathit{A}\\ =\Delta {\mathit{A}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}\Delta \mathit{A}+2\Delta {\mathit{A}}^{\mathrm{T}}\left({\mathbf{I}}_{\left(2N_S+1\right)\times n}\otimes \Delta {\mathit{R}}^{\mathrm{T}}\right){\mathit{h}}_{\mathrm{mc}}\Delta \mathit{A}\\ =\Delta {\mathit{A}}^{\mathrm{T}}\left({\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}+2\Delta {\mathit{A}}^{\mathrm{T}}\left({\displaystyle \sum _{i=1}^{\left(2N_S+1\right)\times n}\mathit{R}\left(i,:\right){\mathit{h}}_{\mathrm{mc}}^{\left(i,j\right)}}\right)\right)\Delta \mathit{A}.\end{array}$$

(20)

Here, $h_{\mathrm{mc}}^{\left(i,j\right)}$ in Equation (20) denotes the integral, over one vibration period, of the product of the second-derivative matrix of the $i$-th degree-of-freedom equation and the $j$-th harmonic term. It can be expressed as:

$${\mathit{h}}_{\mathrm{mc}}^{\left(i,j\right)}={\displaystyle \frac{1}{2}}{\displaystyle {\int }_0^{2\pi }\mathit{S}\left(i,\left(i-1\right)\times \left(2N_s+1\right)+j\right)}{\mathit{S}}^{\mathrm{T}}{\displaystyle \frac{{\mathrm{d}}^2{G}_i}{\mathrm{d}{q}^2}}\mathit{S}d\tau .$$

(21)

To simplify the computation, $h_{\mathrm{mc}}^{\left(i,j\right)}$ is often set identically to zero, which leads to the Gauss–Newton (GN) iteration step $\Delta {\mathit{A}}_{\mathrm{GN}}$:

$$\Delta {\mathit{A}}_{\mathrm{GN}}=-{\left[{\mathit{G}}_{\mathrm{GN}}\left(\mathit{A}\right)\right]}^{-1}{\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}\mathit{R}=-{\left[{\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}\right]}^{-1}{\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}\mathit{R}.$$

(22)

When ${\mathit{K}}_{mc}$ is invertible, Equation (23) can be further simplified as:

$$\Delta \mathit{A}=-{\mathit{K}}_{\mathrm{mc}}^{-1}\mathit{R},$$

(23)

which is equivalent to the search direction of the IHB method. According to Equation (20), the system gradient $\mathit{g}$ and the Newton (NR) iteration step $\Delta {\mathit{A}}_{\mathrm{NR}}$ are respectively given by:

$$\begin{array}{l}\mathit{g}=\nabla f\left(\mathit{A}\right)=\nabla \left({\displaystyle \frac{1}{2}}{‖\mathit{R}‖}^2\right)={\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}\mathit{R},\\ \Delta {\mathit{A}}_{\mathrm{NR}}=-{\left[{\mathit{G}}_{\mathrm{NR}}\left(\mathit{A}\right)\right]}^{-1}\mathit{g}=-{\left[{\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}+{\mathit{H}}_S\right]}^{-1}{\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}\mathit{R},\end{array}$$

(24)

where

$${\mathit{H}}_S=2{\displaystyle \sum _{i=1}^{\left(2N_S+1\right)\times n}\mathit{R}\left(i,:\right){\mathit{h}}_{\mathrm{mc}}^{\left(i,j\right)}}.$$

3.2. Fast Fourier Transform (FFT)

The relevant system functions can be regarded as $2\pi$-periodic functions. Using the Fourier series, the residual function $\mathit{f}\mathit{R}$ can be rewritten as:

$$\mathit{f}\mathit{R}=\mathit{S}\mathit{\psi },$$

(25)

where $\mathit{\psi }$ is the Fourier coefficient vector of the residual function $\mathit{f}\mathit{R}$ and is a constant vector. Owing to the orthogonality of trigonometric functions, the residual vector matrix $\mathit{R}$ can be expressed as:

$$\begin{array}{l}\mathit{R}={\displaystyle {\int }_0^{2\pi }{\mathit{S}}^{\mathrm{T}}}\mathit{S}\mathit{\psi }\mathrm{d}\tau ={\displaystyle {\int }_0^{2\pi }\mathrm{diag}\left(C_S^{\mathrm{T}}{C}_S^{\mathrm{T}},\dots \dots ,C_S^{\mathrm{T}}\right)}\mathrm{d}\tau \mathit{\psi }\\ =\mathrm{diag}\left(\mathrm{diag}\left(2,1,\dots \dots 1\right),\mathrm{diag}\left(2,1,\dots \dots 1\right)\dots \dots ,\mathrm{diag}\left(2,1,\dots \dots 1\right)\right)\mathit{\psi }\\ =\mathrm{diag}\left(D,D,\dots \dots ,D\right)\mathit{\psi }.\end{array}$$

(26)

To efficiently obtain the Fourier coefficient vector $\mathit{\psi }$, uniform discrete sampling is performed over the period $T=2\pi$. Let the number of sampling points be $M$, satisfying $M\ge 2N+1$; $M$ is typically chosen as a power of 2 to take advantage of the efficiency of the FFT. The sampling points are:

$${\tau }_j={\displaystyle \frac{2\pi j}{M}},j=0,1,\dots \dots ,M-1.$$

(27)

For a given coefficient vector $\mathit{A}$, the displacement sequence of each degree of freedom can be obtained via the inverse discrete Fourier transform (IDFT). For the $i$-th degree-of-freedom equation, its complex array $X_i\left[k\right]$ is given by:

$$\begin{array}{l}{X}_i\left[0\right]=a_{i0},\\ X_i\left[k\right]={\displaystyle \frac{a_{ik}-i{b}_{ik}}{2}},k=1,2,\dots \dots N_S,\\ X_i\left[M-k\right]={\displaystyle \frac{a_{ik}+i{b}_{ik}}{2}},k=1,2,\dots \dots N_S,\\ X_i\left[k\right]=0,k>M\mathrm{or}k<0.\end{array}$$

(28)

Applying the inverse Fourier transform to $\left\{X_i\left[k\right]\right\}$ yields the displacement sequence $q_i\left({\tau }_j\right)$. On this basis, multiplying $X_i\left[k\right]$ by $ik\omega$ and $-k^2{\omega }^2$ gives the corresponding velocity sequence $u_i\left({\tau }_j\right)$ and acceleration sequence. Using the above data, the Fourier transform is applied to the residual sequence of each degree of freedom, i.e.,

$${\tilde{F}}_i={\displaystyle \sum _{j=0}^{M-1}f{R}_i}\left({\tau }_j\right)\exp \left(-i{\displaystyle \frac{2\pi jk}{M}}\right),k=0,1,\dots \dots ,M-1,$$

(29)

where ${\tilde{F}}_i={\tilde{F}}_i^{\mathrm{Re}}\left(k\right)+i{\tilde{F}}_i^{\mathrm{Im}}\left(k\right)$. Normalizing ${\tilde{F}}_i$ gives:

$${\tilde{f}}_i\left(k\right)=\left\{\begin{array}{c}{\displaystyle \frac{{\tilde{F}}_i\left(0\right)}{M}},k=1,2,\dots \dots M,\\ {\displaystyle \frac{2{\tilde{F}}_i\left(k\right)}{M}},k=1,2,\dots \dots M,\\ 0,\mathrm{other}.\hfill \end{array}\right.$$

(30)

Solving Equations (27)–(30) jointly, the Fourier coefficient vector of the system, $\mathit{\psi }$, can be expressed as:

$$\mathit{\psi }=\left[{\mathit{\psi }}_1,{\mathit{\psi }}_2,\dots \dots ,{\mathit{\psi }}_n\right],$$

(31)

where

$${\mathit{\psi }}_{i,0}={\tilde{f}}_i^{\mathrm{Re}}\left(0\right),{\mathit{\psi }}_{i,k}^C={\tilde{f}}_i^{\mathrm{Re}}\left(k\right),k=1,2,\dots \dots N_S,{\mathit{\psi }}_{i,k}^S={\tilde{f}}_i^{\mathrm{Im}}\left(k\right),k=1,2,\dots \dots N_S.$$

Using the obtained Fourier coefficients $\mathit{\psi }$, the residual vector of the $i$-th degree of freedom, ${\mathit{R}}_i$, can be written as:

$${\mathit{R}}_i=\mathit{D}\mathit{\psi },$$

(32)

where

$$R_{i,0}=2{\mathrm{\Psi }}_{i,0},R_{i,k}^C={\mathrm{\Psi }}_{i,k}^C,k=1,2,\dots \dots N_s,R_{i,k}^S={\mathrm{\Psi }}_{i,k}^S,k=1,2,\dots \dots N_s.$$

The system Jacobian matrix ${\mathit{K}}_{mc}$ can be expressed as:

$${\mathit{K}}_{\mathrm{mc}}=\left(\begin{array}{ccc}{\displaystyle \frac{\partial R_1}{\partial a_{1,0}}}& \dots & {\displaystyle \frac{\partial R_1}{\partial b_{n,N_S}}}\\ ⋮& \ddots & ⋮\\ {\displaystyle \frac{\partial R_n}{\partial a_{1,0}}}& \dots & {\displaystyle \frac{\partial R_n}{\partial b_{n,N_S}}}\end{array}\right),$$

(33)

where

$$\begin{array}{c}{\mathit{K}}_{mc}\left(j,:\right)={\displaystyle {\int }_0^{2\pi }\mathit{S}{\left(j,:\right)}^{\mathrm{T}}}\left({\displaystyle \frac{\partial \mathit{f}\mathit{R}}{\partial {\mathit{q}}_0^{″}}}{\displaystyle \frac{\partial {\mathit{q}}_0^{″}}{\partial \mathit{A}}}+{\displaystyle \frac{\partial \mathit{f}\mathit{R}}{\partial {\mathit{q}}_0^{\prime }}}{\displaystyle \frac{\partial {\mathit{q}}_0^{\prime }}{\partial \mathit{A}}}+{\displaystyle \frac{\partial \mathit{f}\mathit{R}}{\partial {\mathit{q}}_0}}{\displaystyle \frac{\partial {\mathit{q}}_0}{\partial \mathit{A}}}\right)\\ ={\displaystyle {\int }_0^{2\pi }\mathit{S}{\left(j,:\right)}^{\mathrm{T}}}\left({\omega }_0^2\mathit{M}{\displaystyle \frac{d^2\mathit{S}}{d{\tau }^2}}+{\omega }_0\mathit{C}{\displaystyle \frac{d\mathit{S}}{d\tau }}+\left(\mathit{K}+{\mathit{K}}_{\mathrm{nonlinear}}\right)\mathit{S}\right).\end{array}$$

Based on Equation (33) and using Equations (27)–(31), the system Jacobian matrix can be readily obtained. By repeating the above procedure, ${\mathit{H}}_{\mathrm{mc}}$ and ${\mathit{R}}_{mc}$ can be rapidly derived once the residual function $\mathit{f}\mathit{R}$ is known.

3.3. Adaptive Adjustment of the Search Direction Within a Trust-Region Framework

The MHTR-IHB method is a hybrid approach that, within the trust-region framework, integrates the Newton step $\Delta {\mathit{A}}_{\mathrm{NR}}$, the Gauss–Newton step $\Delta {\mathit{A}}_{\mathrm{GN}}$, and the negative gradient $\mathit{g}$ as iterative directions. Its Hessian matrix can be written as:

$$\begin{array}{cc}\hfill {\mathit{H}}_{\mathrm{mc}}& =\left(1-\alpha \right){\mathit{G}}_{\mathrm{GN}}+\alpha {\mathit{G}}_{\mathrm{NR}}=\left(1-\alpha \right){\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}+\alpha \left({\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}+{\mathit{H}}_S\right)\hfill \\ & ={\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}+\alpha {\mathit{H}}_S,\hfill \end{array}$$

(34)

where the mixing parameter $\alpha \in \left[0,1\right]$ in the Hessian matrix quantifies the relative weighting between the GN and NR directions. The Cauchy point of the system, $\Delta {\mathit{A}}_{\mathrm{cp}}$, is:

$$\Delta {\mathit{A}}_{\mathrm{cp}}=-{\displaystyle \frac{{‖\mathit{g}‖}^2}{{\mathit{g}}^{\mathrm{T}}{\mathit{H}}_{\mathrm{mc}}\mathit{g}}}\mathit{g}.$$

(35)

Accordingly, under the condition of no trust-region constraint, the search direction of the MHTR-IHB method, $\Delta {\mathit{A}}_{\mathrm{HN}}$, is:

$$\Delta {\mathit{A}}_{\mathrm{HN}}=-{\left[{\mathit{H}}_{\mathrm{mc}}\right]}^{-1}\mathit{g}=-{\left[{\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}+\alpha {\mathit{H}}_S\right]}^{-1}{\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}\mathit{R}.$$

(36)

The MHTR-IHB method controls the iteration step by adjusting the trust-region radius $\Delta$. and the mixing parameter $\alpha$, so as to ensure that the search direction reduces the residual. At the $k$-th iteration, expanding Equation (19) in a Taylor series, the predicted reductions and the actual reduction associated with the GN, NR, and HN directions are respectively given by:

$$\begin{array}{l}{\Delta }_{\mathrm{GN}}={\left({\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}\mathit{R}\right)}^{\mathrm{T}}\Delta {\mathit{A}}_{\mathrm{HN}}+1/2\Delta {\mathit{A}}_{\mathrm{HN}}^{\mathrm{T}}\left({\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}\right)\Delta {\mathit{A}}_{\mathrm{HN}},\\ {\Delta }_{\mathrm{NR}}={\left({\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}\mathit{R}\right)}^{\mathrm{T}}\Delta {\mathit{A}}_{\mathrm{HN}}+1/2\Delta {\mathit{A}}_{\mathrm{HN}}^{\mathrm{T}}\left({\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}+{\mathit{H}}_S\right)\Delta {\mathit{A}}_{\mathrm{HN}},\\ {\Delta }_{\mathrm{HN}}={\left({\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}\mathit{R}\right)}^{\mathrm{T}}\Delta {\mathit{A}}_{\mathrm{HN}}+1/2\Delta {\mathit{A}}_{\mathrm{HN}}^{\mathrm{T}}\left({\mathit{K}}_{\mathrm{mc}}^{\mathrm{T}}{\mathit{K}}_{\mathrm{mc}}+\alpha {\mathit{H}}_S\right)\Delta {\mathit{A}}_{\mathrm{HN}},\\ \Delta f=f\left(A_k+\Delta A_{\mathrm{HN}}\right)-f\left(A_k\right).\end{array}$$

(37)

Three indices, ${\rho }_{\mathrm{GN}}$, ${\rho }_{\mathrm{NR}}$, and ${\rho }_{\mathrm{HN}}$, are introduced to characterize the agreement of $q_{\mathrm{GN}}^{\left(k\right)}$, $q_{\mathrm{NR}}^{\left(k\right)}$, and $q_{\mathrm{HN}}^{\left(k\right)}$ with $f_k$. Accordingly, ${\rho }_{\mathrm{GN}}$, ${\rho }_{\mathrm{NR}}$, and ${\rho }_{\mathrm{HN}}$ are defined as:

$${\rho }_{\mathrm{GN}}={\displaystyle \frac{\Delta f}{{\Delta }_{\mathrm{GN}}}},{\rho }_{\mathrm{NR}}={\displaystyle \frac{\Delta f}{{\Delta }_{\mathrm{NR}}}},{\rho }_{\mathrm{HN}}={\displaystyle \frac{\Delta f}{{\Delta }_{\mathrm{HN}}}}.$$

(38)

In general, the closer ${\rho }_{\mathrm{GN}}$ and ${\rho }_{\mathrm{NR}}$ are to 1, the higher the agreement of $q_{\mathrm{GN}}^{\left(k\right)}$, $q_{\mathrm{NR}}^{\left(k\right)}$, and $q_{\mathrm{HN}}^{\left(k\right)}$ with $\Delta f$. According to the trust-region algorithm, the trust-region radius is $\Delta$. When $‖\Delta \mathit{A}‖<\Delta$, $\Delta \mathit{A}=\Delta {\mathit{A}}_{\mathrm{HN}}$. When $‖\Delta {\mathit{A}}_{\mathrm{cp}}‖>\Delta$, $\Delta \mathit{A}=\left({\displaystyle \frac{\Delta }{‖\Delta {\mathit{A}}_{\mathrm{cp}}‖}}\right)\Delta {\mathit{A}}_{\mathrm{cp}}$. When $‖\Delta {\mathit{A}}_{\mathrm{cp}}‖\le \Delta \le ‖\Delta {\mathit{A}}_{\mathrm{HN}}‖$, $\Delta \mathit{A}$ lies on the line segment connecting $‖\Delta {\mathit{A}}_{\mathrm{cp}}‖$ and $‖\Delta {\mathit{A}}_{\mathrm{NH}}‖$, with $\Delta \mathit{A}=\Delta$. The trust-region radius is $\Delta$ updated according to the ratio ${\rho }_{\mathrm{HN}}$, defined as the ratio of actual reduction to the predicted reduction. Specifically, if ${\rho }_{\mathrm{HN}}\ge 0.75$, the radius is expanded to $2\Delta$ to accelerate convergence; if $0.5<{\rho }_{\mathrm{HN}}<0.75$, the radius remains unchanged; if $0.1<{\rho }_{\mathrm{HN}}\le 0.5$, the radius is reduced to $0.5\Delta$; and if ${\rho }_{\mathrm{HN}}\le 0.1$, the radius is also reduced to $0.5\Delta$, and the step is rejected and recomputed. The acceptance of a trial step is governed by the same criterion: a step is accepted only when ${\rho }_{\mathrm{HN}}>0.1$; otherwise, it is rejected, and the trust-region radius is contracted before a new step is computed. To prevent the system from being trapped in a local minimum, a stagnation parameter is introduced, denoted by ${\gamma }_f$:

$$r_f={\displaystyle \frac{\Delta f}{f\left(\mathit{A}\right)}}.$$

(39)

When stagnation is detected, the step size is too small. A jump–backtracking step is then executed by alternating with the parity (odd/even) of the control counter $n_2$. The detailed procedures for determining the search direction bias (Algorithm A1), updating strategy for $\alpha$ (Algorithm A2), and updating the trust-region radius together with the jump–backtracking strategy (Algorithm A3) are provided in Appendix A.

The computational architecture of the proposed method is illustrated in Figure 1. It is formed by coupling an adaptive arc-length continuation module with the MHTR-IHB solution core. The overall solution procedure is divided into two stages. The first stage aims to obtain four initial reference solutions for initiating the continuation. Starting from these, the second stage completes automated tracking of the entire frequency response curve through an iterative predictor–corrector loop.

4. Numerical Examples

In this section, three different nonlinear systems are employed to validate the performance of the proposed MHTR-IHB algorithm, including a generalized Van der Pol oscillator under parametric excitation, an axially moving system, and a composite hysteretic–Duffing oscillator system. Comparisons are made among the fourth-order Runge–Kutta (RK) method, the IHB method, the TR-IHB method, and the MHTR-IHB method in terms of four aspects: the extent of the convergence basin, convergence time, number of iterations, and convergence path.

4.1. Generalized van der Pol Oscillator Under Parametric Excitation

The generalized Van der Pol oscillator considered in this paper incorporates self-excited oscillation, parametric excitation, and the Duffing oscillator equation. The dynamic model of the system is shown in Figure 2.

Owing to its complexity, this system is commonly used to benchmark the effectiveness of numerical algorithms. Its nonlinear equation can be written as [35,36]:

$$m{\displaystyle \frac{{\mathrm{d}}^2{q}_0}{\mathrm{d}{t}_1^2}}-(c_1-c_2{q}_0^2){\displaystyle \frac{\mathrm{d}{q}_0}{\mathrm{d}{t}_1}}+\left(k_1+k_2\cos \left(2{\omega }_1{t}_1\right)\right)q_0+k_3{q}_0^3=0,$$

(40)

where $c_1$ and $c_2$ are the linear and nonlinear damping coefficients, respectively; $k_1$ and $k_2$ are the linear and nonlinear stiffness coefficients, respectively; and $k_2$ and ${\omega }_1$ are the amplitude and frequency of the parametric excitation. Introducing a characteristic length $L$ (for simplicity, $L=1m$ is typically adopted), the above equation is nondimensionalized as:

$${\omega }_0^2={\displaystyle \frac{k_1}{m}},\omega ={\displaystyle \frac{{\omega }_1}{{\omega }_0}},{\alpha }_1=c_1\sqrt{{\displaystyle \frac{1}{m{k}_1}}},{\alpha }_2=c_2{L}^2\sqrt{{\displaystyle \frac{1}{m{k}_1}}},\beta ={\displaystyle \frac{k_2}{k_1}},\gamma ={\displaystyle \frac{k_3{L}^2}{k_1}},\tau ={\omega }_0{\mathrm{T}}_1.$$

In this paper, the parameters are set as ${\alpha }_1=1,a_2=1,\beta =4,\gamma =1,\omega =1$. Equation (41) then becomes:

$${\omega }^2{\displaystyle \frac{d^{2}q}{d{\tau }^2}}-\omega \left({\alpha }_1-{\alpha }_2{q}^2\right){\displaystyle \frac{dq}{d\tau }}+\left(1+\beta \cos \left(2\tau \right)\right)q+\gamma q^3=0.$$

(41)

A periodic solution of Equation (41) takes the form:

$$q\left(\tau \right)={\displaystyle \sum _{i=1}^7{a}_{2i-1}\cos \left(2i-1\right)\tau }+b_{2i-1}\sin \left(2i-1\right)\tau .$$

(42)

To verify the numerical accuracy of the MHTR-IHB method, Figure 3 presents the periodic response of the generalized Van der Pol oscillator at $\omega =1$. The time histories and phase portraits obtained by the proposed method coincide with those computed by the fourth-order RK method and the IHB method. This confirms that MHTR-IHB retains the high-accuracy advantage of IHB while not sacrificing computational accuracy due to the introduction of the symmetric hybrid Hessian matrix and the trust-region framework.

To quantitatively evaluate the sensitivity of the algorithms to the initial guess, a systematic scanning test was performed in a two-dimensional parameter space spanned by the dominant harmonic coefficients $a_1$ and $a_3$, with all other harmonic terms set to zero (domain $\left[-5,5\right]\times \left[-5,5\right]$, 2500 grid points). The resulting convergence basins are shown in Figure 4. For the conventional IHB method (Figure 4a), the convergent initial points are highly concentrated; the method has difficulty converging from initial guesses with large residuals and often falls into the trivial zero solution. In contrast, the convergent points of MHTR-IHB (Figure 4b) are distributed throughout the entire scanned region, demonstrating robust convergence to the true physical solution under a wide range of initial conditions. This indicates that, through the combined control of the iteration step size by the symmetric hybrid Hessian matrix and the trust-region algorithm, MHTR-IHB avoids erroneous iterative trajectories and effectively enlarges the basin of convergence.

The iteration details (Figure 5) further reveal distinct convergence mechanisms of the two methods. For IHB, the residual norm and the solution norm decay simultaneously, which is a typical signature of convergence to the trivial zero solution. By contrast, the proposed MHTR-IHB method exhibits a more intricate adaptive behavior: the mixing parameter α is not fixed but is dynamically adjusted according to the iteration state, and its full evolution history (Figure 5b) governs the intelligent progression of the search direction. This intrinsic adaptivity enables the algorithm to maintain the correct convergence path under the trust-region framework (Figure 5c), effectively distinguishing between “residual reduction” and “convergence to the physical solution” and thereby ensuring the physical reliability of the computed results.

The parametric frequency response curves shown in Figure 6 comprehensively demonstrate the overall performance of the proposed algorithm. Compared with the RK method, which can capture only part of the stable solutions, MHTR-IHB, like IHB, can obtain both stable and unstable periodic solutions of the system. Notably, when the parametric excitation intensity is $\beta =2$, the response curve exhibits a closed hysteresis loop (Figure 6), a strongly nonlinear feature that readily causes conventional iterations to fail. Nevertheless, by virtue of the built-in adaptive hybrid search-direction mechanism, MHTR-IHB intelligently adjusts the step size within the trust-region framework and successfully traverses these numerically sensitive regions. The efficiency comparison (

Table 1. Summary results for the frequency–amplitude response curves.

Method	$\mathit{\beta }$	Points	Total Iterations	Total Computation Time (s)
IHB	4	303	695	1312
MHTR-IHB	4	278	452	205
IHB	2	299	725	1405
MHTR-IHB	2	265	465	231

) is more direct: MHTR-IHB reduces the number of continuation points required for full-path tracking, and its total computational time is less than 16% of that of the IHB algorithm. This improvement arises because the symmetric hybrid Hessian matrix optimizes the iterative trajectory and reduces redundant iterations, while the use of FFT substantially enhances computational efficiency.

4.2. Axially Moving System

Axially moving structures (e.g., belts and cables) are typical components in mechanical conveying and continuous rolling systems, and their dynamics often exhibit complex coupled effects such as geometric nonlinearity and internal resonance. Taking the simply supported axially moving belt shown in Figure 7 as an example, the model involves geometric and material parameters, including the belt density $\rho$, cross-sectional area A, bending stiffness $D$, axial tension $T$, axial length $L$, and belt width $b$. The system is also subjected to a velocity-dependent damping coefficient $c_0$, an axial speed $V_0$, an external load $F_0$, and an excitation frequency ${\omega }_0$.

Neglecting the displacement in the $y$-direction, the governing equation of axial motion for the belt and the associated boundary conditions are given by [37,38]:

$$\rho h\left({\displaystyle \frac{{\mathrm{d}}^2{w}_0}{\mathrm{d}{t}_0^2}}+2V{\displaystyle \frac{\mathrm{d}{w}_0}{\mathrm{d}{t}_0}}+\left(V_0^2-1\right){\displaystyle \frac{{\mathrm{d}}^2{w}_0}{\mathrm{d}{x}^2}}\right)+D{\displaystyle \frac{{\mathrm{d}}^4{w}_0}{\mathrm{d}{x}^4}}+c{\displaystyle \frac{\mathrm{d}{w}_0}{\mathrm{d}{t}_0}}=F_0\cos \left({\omega }_0{t}_0\right).$$

(43)

Introduce dimensionless parameters and nondimensionalize the system, i.e.,

$$w={\displaystyle \frac{w_0}{L}},\eta ={\displaystyle \frac{L}{b}},\epsilon ={\displaystyle \frac{D}{T{L}^2}},F={\displaystyle \frac{F_0}{T{L}^2}};t=t_0\sqrt{{\displaystyle \frac{T}{\rho A{L}^2}}},V=V_0\sqrt{{\displaystyle \frac{\rho A}{T}}},\omega ={\omega }_0\sqrt{{\displaystyle \frac{\rho h{L}^2}{T}}},c=c_0{\displaystyle \frac{L}{\sqrt{\rho AT}}}.$$

Using the simply supported boundary conditions of the belt and separating the spatial variable $\eta$ and the time variable $\tau$ via the Galerkin method, one obtains a nonlinear equation of motion represented by two transverse modal displacement components $\mathit{q}={\left[q_1,q_2\right]}^{\mathrm{T}}$:

$$\mathit{M}{\displaystyle \frac{{\mathrm{d}}^2\mathit{q}}{d{t}^2}}+\mathit{C}{\displaystyle \frac{\mathrm{d}\mathit{q}}{dt}}+\left(\mathit{K}+{\mathit{K}}_3\right)\mathit{q}=\mathit{f}\cos (\omega t),$$

(44)

where the external load vector $\mathit{f}$, mass matrix $\mathit{M}$, damping matrix $\mathit{C}$, stiffness matrix $\mathit{K}$, and nonlinear stiffness matrix ${\mathit{K}}_3$ are given as follows:

$$\begin{array}{l}\mathit{M}=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right],\mathit{C}=\left[\begin{array}{cc}c& -{\displaystyle \frac{16}{3}}V\\ {\displaystyle \frac{16}{3}}V& c\end{array}\right],\mathit{K}=\left[\begin{array}{cc}{\omega }_1^2& 0\\ 0& {\omega }_2^2\end{array}\right],\\ {\mathit{K}}_3=\left[\begin{array}{cc}{k}_{11}{q}_1^2+{\displaystyle \frac{1}{3}}{k}_{12}{q}_2^2& {\displaystyle \frac{2}{3}}{k}_{12}{q}_1{q}_2\\ {\displaystyle \frac{2}{3}}{k}_{21}{q}_1{q}_2& k_{22}{q}_2^2+{\displaystyle \frac{1}{3}}{k}_{21}{q}_1^2\end{array}\right],f=[f_1;0].\end{array}$$

When the axial speed is $V=0.8$, the parameters in Equation (42) are ${\omega }_{10}^2=42.5132,{\omega }_{20}^2=382.57,c=0.1,k_{11}=22.88,k_{12}=40.11,k_{21}=40.11$, $k_{22}=437.7,f_1=5$. The system exhibits internal resonance in this case, and the natural-frequency ratio is ${\displaystyle \frac{{\omega }_2}{{\omega }_1}}\approx 3$. Let $N=7$; the solution is assumed in the form:

$$q_i\left(\tau \right)={\displaystyle \sum _{k=1}^N{a}_{i,2k-1}\cos \left(2k-1\right)\tau +b_{i,2k-1}\sin \left(2k-1\right)\tau },i=1,2.$$

(45)

At the excitation frequency $\omega =8$, Figure 8 compares the periodic responses computed by the RK method, the IHB method, and the MHTR-IHB method. The three results coincide completely in both the time histories and phase portraits, confirming that MHTR-IHB retains high accuracy under internal-resonance conditions.

To evaluate the sensitivity of the algorithm to the initial guess, systematic scans of the convergence basin were conducted in the parameter plane spanned by the key harmonic coefficients $a_{1,1}$ and $b_{1,1}$ (domain $\left[-10,10\right]\times \left[-10,10\right]$). The results (Figure 9) clearly show that the convergent initial points of the conventional IHB method (Figure 9a) are highly clustered in a narrow region with small initial residuals, indicating strong dependence on the initial values. In contrast, the convergent points of MHTR-IHB (Figure 9c) are nearly distributed over the entire test domain, yielding a substantially enlarged basin of convergence. In particular, Figure 9b provides, for comparison, the convergence basin of the TR-IHB method, which employs the trust-region framework but uses a fixed Gauss–Newton direction (i.e., $\alpha =0$). Its convergence is better than that of IHB but remains markedly inferior to that of MHTR-IHB, directly demonstrating the necessity and superiority of introducing a symmetric hybrid Hessian matrix (with dynamically varying $\alpha$).

Figure 10 further illustrates the iterative process for a large-residual initial point ($a_{1,1}=-10,b_{1,1}=-8.36$). The IHB algorithm diverges completely. The TR-IHB method (Figure 10a), although employing a jump–backtracking strategy under trust-region control, still fails to converge, indicating that step-size control alone cannot overcome the problem caused by an incorrect search direction. In contrast, MHTR-IHB (Figure 10b) successfully guides the iteration to convergence by dynamically adjusting the mixing parameter $\alpha$ and flexibly switching the search direction.

Finally, Figure 11 presents the complete frequency response curves of the system. Unlike the RK method, which can obtain only stable steady-state periodic solutions, the MHTR-IHB method not only efficiently tracks the entire set of solution branches, including unstable branches (dashed lines), but also clearly reveals multiple jump points such as QS1 and QS2 induced by internal resonance, as well as a secondary resonance peak, thereby fully demonstrating its comprehensive advantages in analyzing strongly nonlinear, multi-branch systems. The efficiency comparison (Table 2) shows that, while reducing both the number of sampling points and the number of iterations, the computational time of MHTR-IHB is further reduced to 18% of that of the conventional IHB method, mainly due to the acceleration provided by the FFT algorithm. These results jointly confirm that the proposed method exhibits pronounced advantages in both robustness and computational efficiency when dealing with strongly nonlinear systems.

Figure 11. Frequency–amplitude response curves.

Figure 11. Frequency–amplitude response curves.

Table 2. Computational parameters for the frequency–amplitude response curves in Figure 12.

	Points	Total Iterations	Total Computation Time (s)
IHB	795	1908	3542
MHTR-IHB	678	1356	635

Table 2. Computational parameters for the frequency–amplitude response curves in Figure 12.

	Points	Total Iterations	Total Computation Time (s)
IHB	795	1908	3542
MHTR-IHB	678	1356	635

Figure 12. Schematic of the rolling system model.

Figure 12. Schematic of the rolling system model.

4.3. Periodic Response of a Composite Hysteretic–Duffing Oscillator

To validate the performance of the algorithm under extremely complex operating conditions, this subsection constructs and analyzes a nonlinear hysteretic dynamic model of the work roll during the rolling process. The schematic of the dynamic model is shown in Figure 12.

This model represents, in parallel, a Bouc–Wen element and a cubic stiffness term, thereby simultaneously capturing the hysteretic effect present in the rolling zone and the geometrically nonlinear hardening effect, and it is subjected to multi-frequency excitation. The governing equations can be expressed as [39]:

$$\begin{array}{l}m{\displaystyle \frac{{\mathrm{d}}^{2}x}{\mathrm{d}{t}_1^2}}+c{\displaystyle \frac{\mathrm{d}z}{\mathrm{d}{t}_1}}+\alpha k_{1}x+k_{2}z+k_3{x}^3=f_1\cos \left({\omega }_1{t}_1\right)+f_2\cos \left(3{\omega }_1{t}_1\right),\\ {\displaystyle \frac{\mathrm{d}z}{\mathrm{d}{t}_1}}=A{\displaystyle \frac{\mathrm{d}x}{\mathrm{d}{t}_1}}-{\beta }_1\left|{\displaystyle \frac{\mathrm{d}x}{\mathrm{d}{t}_1}}\right|{\left|z\right|}^{n-1}z-{\gamma }_1{\displaystyle \frac{\mathrm{d}x}{\mathrm{d}{t}_1}}{\left|z\right|}^n.\end{array}$$

(46)

Introducing the natural frequency ${\omega }_0=\sqrt{{\displaystyle \frac{k_1}{m}}}$ and the characteristic length $L$, the dimensionless parameters in Equation (46) are defined as:

$$\begin{array}{c}{q}_1={\displaystyle \frac{x}{L}},q_2={\displaystyle \frac{z}{L}},\omega ={\displaystyle \frac{{\omega }_1}{{\omega }_0}},{\lambda }_1={\displaystyle \frac{k_2}{m{\omega }_0^2}},{\lambda }_1={\displaystyle \frac{k_3}{m{\omega }_0^2}};\xi ={\displaystyle \frac{c}{\sqrt{k_{1}m}}},\omega t={\omega }_1{t}_1,\\ F_1={\displaystyle \frac{f_1}{k_{1}L}},F_2={\displaystyle \frac{f_2}{k_{1}L}},\beta ={\beta }_1{L}^n,\gamma ={\gamma }_1{L}^n.\end{array}$$

The dimensionless governing equation of the work roll in Equation (46) can then be rewritten as:

$$\mathit{M}{\displaystyle \frac{{\mathrm{d}}^2\mathit{q}}{d{t}^2}}+\mathit{C}{\displaystyle \frac{\mathrm{d}\mathit{q}}{dt}}+\mathit{R}\mathit{F}\left(\mathit{q},{\displaystyle \frac{\mathrm{d}\mathit{q}}{\mathrm{d}t}}\right)=\mathit{F},$$

(47)

where

$$\begin{array}{c}\mathit{M}=\left[\begin{array}{cc}1& 0\\ 0& 0\end{array}\right],\mathit{C}=\left[\begin{array}{cc}\zeta & 0\\ -A& 1\end{array}\right],\mathit{R}\mathit{F}\left(\mathit{q},{\displaystyle \frac{\mathrm{d}\mathit{q}}{\mathrm{d}t}}\right)=\left[\begin{array}{c}\alpha q_1+{\lambda }_1{q}_2+{\lambda }_2{q}_1^3\\ \beta \left|{\displaystyle \frac{\mathrm{d}{q}_1}{\mathrm{d}t}}\right|{\left|q_1\right|}^{n-1}{q}_2+\gamma {\left|q_2\right|}^n{\displaystyle \frac{\mathrm{d}{q}_1}{\mathrm{d}t}}\end{array}\right],\\ \mathit{F}={\left[F_1\cos \left(\omega t\right)+F_2\cos \left(3\omega t\right);0\right]}^{\mathrm{T}},\mathit{q}={\left[q_1;q_2\right]}^{\mathrm{T}}.\end{array}$$

For the above work-roll model, the parameters are set as $\beta =5,\gamma =5,n=1$. The system also includes a linear spring with stiffness $\alpha =1.667$ and a viscous damper with damping coefficient $\zeta =0.04$. The parameters ${\lambda }_1,{\lambda }_2$ are used to regulate the restoring force provided by $q_2$, with ${\lambda }_1=98.33,{\lambda }_2=3.$ The excitation force amplitudes are $F_1=3.6,F_2=1.8$. The hysteretic force in the equation has the magnitude:

$$\mathit{R}\mathit{F}\left(1,1\right)=\alpha q_1+{\lambda }_1{q}_2+{\lambda }_2{q}_1^3.$$

(48)

Let the harmonic truncation order be $N=11$. The periodic solution can be expressed as:

$$q_i\left(\tau \right)={\displaystyle \sum _{k=1}^N{a}_{i,k-1}\cos \left(k-1\right)\tau +b_{i,k-1}\sin k\tau },i=1,2.$$

(49)

Due to the influence of the hysteretic force $\mathit{R}\mathit{F}$, when the excitation frequency is $\omega =3.06$, the velocity and displacement of $q_1$ undergo multiple reversals within one period. As a result, multiple inner loops appear in the phase portrait, and the time history exhibits turning points between successive displacement maxima, as shown in Figure 13.

Corresponding to the steady-state response in Figure 13, Figure 14 presents the system hysteresis loop and the hysteretic-force time history, respectively. The double inner-loop structure of the hysteresis loop (Figure 14a) directly corresponds to the double inner loops in the phase portrait (Figure 13a), indicating that the hysteretic force undergoes multiple loading–unloading cycles within one vibration period, thereby reflecting the energy-dissipation mechanism of this class of systems. By comparing the time histories obtained by MHTR-IHB (Figure 13b and Figure 14b) with those computed by the fourth-order RK method and the conventional IHB method, good agreement among the three results is observed. To further investigate the influence of damping on hysteretic energy dissipation, we computed the steady-state hysteresis loops for three damping coefficients $\left(\zeta =0.04,0.5,3\right)$, as shown in Figure 14c. The area enclosed by the hysteresis loop decreases progressively as the damping increases. This trend can be attributed to nonlinear coupling: larger damping reduces the response amplitude, and the hysteretic energy dissipation exhibits a nonlinear dependence on the amplitude. As a result, the reduction in loop area becomes more pronounced at higher damping levels. These results indicate that, in strongly nonlinear systems, viscous damping can affect hysteretic dissipation indirectly by modulating the response amplitude.

To evaluate the robustness of the algorithm with respect to the initial values, convergence-basin tests were conducted in the plane of the initial harmonic coefficients $a_{1,1}$ and $b_{1,3}$ (with the remaining coefficients set to zero) over $\left[-10,10\right]\times \left[-10,10\right]$. For MHTR-IHB, the nonconvergent points are sparsely distributed within the convergence basin, as shown in Figure 15a. For TR-IHB, because the second-derivative matrix ${\mathit{H}}_s$ is not corrected via the mixing parameter $\alpha$, the convergence basin is further reduced relative to Figure 15a, as shown in Figure 15b. For the IHB method, the convergent points are concentrated in the small-residual region, while in the large-residual case, the convergent points are sparsely distributed.

Figure 16a,b show, for a large-residual initial guess $\left(a_{1,1}=-2.75,b_{1,3}=-5\right)$, the evolution of the residual norm and the mixing parameter α during the iterative processes of the three methods. The conventional IHB method diverges rapidly. The TR-IHB method (with a fixed Gauss–Newton direction) converges after multiple jump–backtracking adjustments, but the process is oscillatory and requires many iterations. By contrast, the proposed MHTR-IHB method dynamically adjusts the search direction via the symmetric hybrid Hessian matrix (with an intelligent evolution of $\alpha$, see Figure 16b); only one strategy adjustment is needed, and robust convergence is achieved within 32 iterations. Moreover, the residual-decay curve is smoother, clearly demonstrating its overall advantages in convergence speed and numerical stability.

In summary, by integrating a trust-region framework with a hybrid Hessian matrix, MHTR-IHB achieves coordinated optimization of the search direction and step size, effectively enhancing the algorithm’s global optimization capability and convergence stability for strongly nonlinear problems. As a result, it markedly improves computational efficiency and robustness while maintaining accuracy.

5. Conclusions

This paper presents a symmetric hybrid Hessian trust-region incremental harmonic balance method (MHTR-IHB) for efficient and robust computation of periodic responses in strongly nonlinear systems. By reformulating the conventional IHB iteration as a nonlinear least-squares optimization problem, the proposed method integrates a trust-region framework with an adaptively blended Hessian matrix that combines Gauss–Newton and Newton directions. This design enables coordinated adjustment of the search direction and step size during the iterative process.

The main findings are summarized as follows:

• Improved convergence robustness. The adaptive hybrid Hessian strategy enlarges the basin of convergence and reduces sensitivity to initial guesses. In the considered examples, the method converges to physically meaningful solutions even for large-residual initial conditions and in the vicinity of trivial solutions.
• Higher computational efficiency. The combination of an optimized iterative trajectory and FFT-based acceleration reduces both the number of iterations and the number of continuation points required for tracing frequency–response curves, while maintaining numerical accuracy.
• Capability for complex nonlinear behaviors. The proposed framework captures a wide spectrum of complex dynamical behaviors, including self-excitation, internal resonance, and hysteretic effects. It can track complete frequency–response curves, including stable and unstable periodic solutions, as well as multiple branches and jump phenomena.

By framing the harmonic balance iteration within an optimization context, the MHTR-IHB method provides a powerful tool for nonlinear vibration analysis and establishes a new paradigm for integrating advanced optimization techniques into frequency-domain methods. Its modular and generalizable structure makes it readily applicable to a broad class of engineering systems. Future work will focus on extending this framework to tackle more challenging problems, such as quasi-periodic responses, bifurcation analysis, and systems with strong nonlinearities and time delays.