A Neural Network-Assisted Variable Step-Size NLMS Algorithm

Abstract

The traditional normalized least-mean-square (NLMS) algorithm faces an inherent trade-off between convergence rate and steady-state error, and its adaptability is limited in non-stationary environments. This paper proposes a neural network-assisted variable step-size NLMS algorithm (NN-VSS-NLMS). An analytically motivated reference step size is first derived under a zero-mean statistically symmetric signal assumption to characterize the desired step-size trend. Based on this reference, an eight-dimensional feature vector composed of input signal power, error energy, and related statistical descriptors is constructed to describe the instantaneous signal state, and a two-layer fully connected neural network (NN) is introduced as an auxiliary tool to provide data-driven correction to the reference step size. In addition, dynamic modulation, step-size constraints, and smoothing operations are incorporated to regulate the predicted step size and enhance its controllability under time-varying conditions. Through simulations with stationary and non-stationary inputs as well as time-invariant and time-varying systems, the proposed algorithm achieves up to a fourfold improvement in convergence rate and more than 8 dB reduction in steady-state error compared with the classical NLMS algorithm, while maintaining improved tracking ability.

1. Introduction

An adaptive filter plays an important role in many signal processing applications [1,2,3,4,5,6], such as echo cancelation, line enhancement, channel equalization, system identification, and time delay estimation [7]. Among adaptive algorithms, the least-mean-square (LMS) algorithm [8,9] is one of the most classical adaptive algorithms due to its simple structure and easy implementation, but its convergence rate strongly depends on the input signal power, which leads to a significant performance degradation when the input amplitude varies widely or exhibits strong correlation [10]. To mitigate this problem, the normalized least-mean-square (NLMS) algorithm [11,12] normalizes the input signal power on the basis of LMS, effectively solving the problem that LMS is sensitive to the input amplitude, and improving the adaptability of the algorithm to input signals with different amplitudes to some extent. However, the step size in NLMS is a fixed value that must be manually set, making it difficult to coordinate convergence rate and steady-state error [13]: a large step size leads to fast convergence but may cause oscillation, whereas a small step size provides better steady-state performance but results in slow convergence [14]. From a system perspective, adaptive filtering inherently involves regulating this convergence–stability trade-off, particularly when signal statistics vary over time. In practical scenarios, signals and environments are often non-stationary, such as time-varying fading in communication channels, dynamic amplitude changes in speech signals, and abrupt variations in noise statistics, and under such conditions, the limitations of a fixed-step-size NLMS algorithm become more pronounced [15].

In recent years, researchers have proposed various variable step-size NLMS (VSS-NLMS) algorithms, enabling the step size to adapt to changes in signal statistics. These algorithms can be broadly divided into three categories. The first category includes step-size adjustment methods driven by error statistics, which directly use error power or noise power to construct step-size update rules. For example, the nonparametric VSS-NLMS (NP-VSS-NLMS) algorithm proposed by Benesty et al. [16] adjusts the step size dynamically according to the ratio between error power and noise power. The VSS-NLMS algorithm proposed by Huang et al. [17] controls the step-size update using the mean-square error (MSE) and the estimated system noise power. The new and effective nonparametric VSS-NLMS algorithm proposed by Wang and Zhang [18] obtains the step size from the square roots of the estimated error power and system noise power. The second category consists of structure-enhanced methods based on statistical inference or optimization theory, which improve step-size adaptability by introducing additional inference mechanisms or optimization frameworks. The EM-NLMS algorithm proposed by Huemmer et al. [19] obtains a variable step size by estimating the state and observation noise variances in real time through the M-step of the EM algorithm. The partial-update NLMS (PU-NLMS) algorithm proposed by Bhotto and Antoniou [20] computes a variable step size by solving a constrained minimization problem and significantly reduces the steady-state misalignment. The step-size optimization NLMS algorithm proposed by Hayashi and Sayed [21] unfolds the NLMS iterations into a differentiable computation graph and trains the step-size parameters directly using an end-to-end optimization framework. The third category adjusts the effective step size from a regularization perspective, where a time-varying regularization term is introduced to control the step-size magnitude equivalently. For example, the generalized squared-error regularized LMS (GSER-LMS) algorithm proposed by Huang and Lee [22] uses the inverse of the weighted squared error as a time-varying regularization parameter to adjust the effective step size dynamically, thereby achieving a balance between fast convergence and steady-state error.

Existing VSS-NLMS algorithms have achieved progress in these aspects, but their step-size adjustment mechanisms are still predominantly based on explicitly designed rules, which are typically derived under stationary or quasi-stationary assumptions. As a result, these methods often exhibit sluggish response or severe step-size oscillations when the signal statistics change abruptly, and their performance degrades significantly in highly non-stationary environments. Moreover, most VSS-NLMS algorithms rely on multiple manually tuned hyperparameters whose optimal values strongly depend on the signal type, signal-to-noise ratio (SNR) level, and system dynamics, thereby limiting their practical applicability and robustness. Although several analytically motivated step-size expressions have been reported, their practical performance is frequently compromised by modeling mismatch, noisy instantaneous estimates, and violated assumptions in real-world scenarios.

More recently, learning-assisted adaptive filtering methods have attracted increasing attention, where neural networks (NNs) or data-driven models are incorporated into classical adaptive filtering frameworks to enhance performance in complex and non-stationary environments [18,21,23]. For instance, NN-augmented adaptive filters have been proposed to learn nonlinear update rules or estimate key parameters such as step size and covariance matrices directly from data [24,25,26]. In addition, meta-learning and deep-learning-based adaptive filtering frameworks have been explored to improve generalization across varying signal conditions [27,28,29]. Despite these advances, most existing learning-assisted approaches either replace the analytical update rule with a fully data-driven model, which reduces interpretability, or rely on loosely coupled combinations of NNs and adaptive filters without a clearly defined analytical reference structure.

These observations indicate that a gap remains between analytically derived VSS methods and learning-assisted approaches. The former provide interpretable structures but are sensitive to modeling assumptions, while the latter improve adaptability but often lack clear analytical grounding. Effectively integrating analytical interpretability with data-driven adaptability remains an open issue.

Motivated by this gap, this work adopts a hybrid learning-assisted framework. Instead of replacing the analytical structure, an explicitly derived reference step size is first constructed based on statistical modeling. From a statistical perspective, the analytical derivation of the reference step size relies on commonly adopted assumptions such as zero-mean Gaussianity and balanced second-order moment structures, which exhibit certain symmetry properties in the signal distributions. Under stationary conditions, these properties facilitate tractable modeling and lead to structured relationships between the input and error signals. However, in practical non-stationary environments, such symmetry conditions may only hold approximately or be partially violated, resulting in biased instantaneous estimates and performance degradation. To address this issue, an NN is introduced as a data-driven correction mechanism to refine the reference step size based on instantaneous signal features and compensate for deviations from the symmetry-based assumptions used in the analytical model, rather than explicitly enforce symmetry. Specifically, the proposed NN-assisted variable step-size NLMS (NN-VSS-NLMS) algorithm constructs an eight-dimensional feature vector to describe the instantaneous features of the signals and uses a two-layer fully connected NN to learn the mapping between the signal features and the required step-size adjustment. Dynamic modulation, steady-state constraint, and smoothing mechanisms are incorporated to further improve the robustness and temporal consistency. The structure of the proposed algorithm is shown in Figure 1. To thoroughly evaluate its performance, the proposed algorithm is compared with classical algorithms under stationary and non-stationary inputs, time-invariant and time-varying systems, and various SNRs.

Compared with existing VSS-NLMS and learning-assisted adaptive filtering methods, the main contributions of this work can be summarized as follows:

• An analytically motivated reference step-size expression is derived based on statistical modeling, providing an interpretable characterization of the desired step-size behavior;
• A NN is introduced as a data-driven correction mechanism to refine the analytical step-size model under non-stationary conditions, rather than replacing it;
• A dynamic modulation and constraint framework is developed to regulate the temporal evolution of the step size and enhance its robustness;
• The proposed method establishes a hybrid model that integrates analytical interpretability with data-driven adaptability.

2. Algorithm Design

This section first reviews the basic principles of the classical NLMS algorithm and then presents the proposed NN-VSS-NLMS algorithm in a systematic manner, including the theoretical derivation of the step size, the NN-based prediction, the dynamic modulation and steady-state constraint mechanisms, and the convergence analysis.

2.1. Review of the NLMS Algorithm

Let the filter length be $L$, and the input vector $\mathit{x}\left(n\right)$ and the coefficient vector $\mathit{w}\left(n\right)$ defined as

$$\mathit{x}\left(n\right)=[\mathit{x}\left(n\right),\mathit{x}\left(n-1\right),\dots ,\mathit{x}\left(n-L+1\right){]}^{\mathrm{T}}$$

(1)

$$\mathit{w}\left(n\right)=[{\mathit{w}}_0\left(n\right), {\mathit{w}}_1\left(n\right), \dots ,{\mathit{w}}_{L-1}\left(n\right){]}^{\mathrm{T}}.$$

(2)

The filter output is

$$y\left(n\right)={\mathit{w}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n\right).$$

(3)

The desired signal is denoted by $d\left(n\right)$ and the error signal $e\left(n\right)$ is defined as

$$e\left(n\right)=d\left(n\right)-y\left(n\right).$$

(4)

The classical LMS algorithm minimizes the MSE through the gradient descent method, and its coefficient update rule is given by

$$\mathit{w}\left(n+1\right)=\mathit{w}\left(n\right)+\mu e\left(n\right)\mathit{x}\left(n\right)$$

(5)

where $\mu$ is the fixed step-size parameter.

To address the instability of coefficient updates caused by variations in the input energy, the NLMS algorithm introduces a normalization term into the update rule

$$\mathit{w}\left(n+1\right)=\mathit{w}\left(n\right)+\mu {\displaystyle \frac{e\left(n\right)\mathit{x}\left(n\right)}{{\parallel \mathit{x}\left(n\right)\parallel }^2+\delta }}$$

(6)

where ${\parallel \mathit{x}\left(n\right)\parallel }^2={\mathit{x}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n\right)$ and $\delta$ is a regularization parameter introduced to avoid division by zero.

By normalizing the input power, the NLMS algorithm effectively adjusts the step size, reducing the update magnitude when the input amplitude is large and preserving a meaningful learning rate when the amplitude is small. In non-stationary environments, the input signal energy and noise power vary over time, which makes the choice of step size more critical. A relatively large step size tends to accelerate the convergence rate but increases the steady-state error, whereas a smaller step size improves steady-state performance at the cost of a slower convergence rate. Under such conditions, variable step size schemes are often adopted to enhance overall performance. The goal is to adjust $\mu$ dynamically according to the current signal or error condition so that a better compromise between convergence rate and steady-state error can be achieved [30].

2.2. Theoretical Derivation of the Reference Step Size

Conventional VSS-NLMS algorithms typically rely on empirically designed step-size rules, whose performance degrades under non-stationary conditions due to their dependence on simplified statistical assumptions. To overcome this limitation, this paper first derives an analytically motivated reference step size, which is subsequently refined using an NN based on instantaneous signal features.

The derivation of the reference step size follows a combination of analytical transformations and commonly adopted modeling approximations in adaptive filtering. The initial steps are based on algebraic manipulations of the NLMS update and the MSE cost function, while subsequent simplifications are introduced to obtain a tractable expression under practical conditions. These simplifications are guided by standard assumptions, including local stationarity within a short-time window, slow variation in the optimal system, approximate projection of the coefficient error onto the input subspace, and weak Gaussianity of the involved random variables. Together, these considerations lead to a reference step-size expression that remains analytically interpretable while being suitable for practical implementation.

In adaptive filtering, the objective is to drive the error $e\left(n\right)=d\left(n\right)-{\mathit{w}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n\right)$ toward zero with a fast convergence rate while preserving stability. Accordingly, the MSE at the next time instant is defined as the cost function:

$$J\left(n+1\right)=E\left[{e\left(n+1\right)}^2\right]$$

(7)

where $E\left[·\right]$ denotes the expectation operator.

The error signal at time $n+1$ is defined as

$$e\left(n+1\right)=d\left(n+1\right)-{\mathit{w}}^{\mathrm{T}}\left(n+1\right)\mathit{x}\left(n+1\right).$$

(8)

Substituting Equation(6) into Equation (8) and rearranging yields

$$e\left(n+1\right)=d\left(n+1\right)-{\mathit{w}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n+1\right)-\mu {\displaystyle \frac{e\left(n\right){\mathit{x}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n+1\right)}{{\parallel \mathit{x}\left(n\right)\parallel }^2+\delta }}.$$

(9)

To simplify the derivation, the two intermediate variables, the residual signal $r\left(n+1\right)$ and the correlation coefficient ${\alpha }_n$, are defined as:

$$r\left(n+1\right)=d\left(n+1\right)-{\mathit{w}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n+1\right).$$

(10)

$${\alpha }_n={\displaystyle \frac{{\mathit{x}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n+1\right)}{{\parallel \mathit{x}\left(n\right)\parallel }^2+\delta }}.$$

(11)

Thus, Equation (9) can be rewritten as

$$e\left(n+1\right)=r\left(n+1\right)-\mu e\left(n\right){\alpha }_n.$$

(12)

Substituting Equation (12) into Equation (7), taking the derivative with respect to $\mu \left(n\right)$, and setting it to zero yields:

$${\displaystyle \frac{\partial J\left(n+1\right)}{\partial \mu \left(n\right)}}={\displaystyle \frac{\partial E\left[{\left(r\left(n+1\right)-\mu e\left(n\right){\alpha }_n\right)}^2\right]}{\partial \mu \left(n\right)}}=0.$$

(13)

This yields the theoretical reference step size as

$$\mu \left(n\right)={\displaystyle \frac{E\left[e\left(n\right){\alpha }_{n}r\left(n+1\right)\right]}{E\left[{e\left(n\right)}^2{{\alpha }_n}^2\right]}}.$$

(14)

The expectation operator is difficult to estimate in real time, especially under non-stationary environments. Therefore, the expectation is approximated using a short-term smoothing operator ${\mathbb{E}}_s\left[·\right]=(1-\lambda ){\sum }_{k=0}^{\infty }{\lambda }^k(·)(n-k)$, where $\lambda \in \left(\mathrm{0,1}\right)$. By assuming that the signal is approximately locally stationary within the smoothing window (meaning that its statistics vary more slowly than the window length), Equation (14) is modified as

$$\mu \left(n\right)={\displaystyle \frac{{\mathbb{E}}_s\left[e\left(n\right){\alpha }_{n}r\left(n+1\right)\right]}{{\mathbb{E}}_s\left[{e\left(n\right)}^2{{\alpha }_n}^2\right]}}.$$

(15)

To further simplify the computation of the reference step size and improve its practical usefulness, it is necessary to expand the residual signal in a mathematical form. Assume that the desired signal satisfies

$$d\left(n\right)={{\mathit{w}}_0}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n\right)+v\left(n\right)$$

(16)

where ${\mathit{w}}_0\left(n\right)$ is the optimal coefficient vector that minimizes the MSE, and $v\left(n\right)$ is zero-mean additive noise uncorrelated with the input signal. Define the coefficient error vector [31] as

$$\tilde{\mathit{w}}\left(n\right)={\mathit{w}}_0\left(n\right)-\mathit{w}\left(n\right).$$

(17)

Substituting $d(n+1)$ into the residual signal in Equation (10), expanding it and approximating ${\mathit{w}}_0(n+1)\approx {\mathit{w}}_0\left(n\right)$ under the assumption of the slow variation in the optimal system, yields

$$\begin{array}{c}r\left(n+1\right)=d\left(n+1\right)-{\mathit{w}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n+1\right)\\ ={{\mathit{w}}_0}^{\mathrm{T}}\left(n+1\right)\mathit{x}\left(n+1\right)+v\left(n+1\right)-{{\mathit{w}}_0}^{\mathrm{T}}(n)\mathit{x}\left(n+1\right)+{\tilde{\mathit{w}}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n+1\right)\\ \approx {\tilde{\mathit{w}}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n+1\right)+v\left(n+1\right).\end{array}$$

(18)

Since the noise $v(n+1)$ is uncorrelated with $e\left(n\right)$ and ${\alpha }_n$, it follows that ${\mathbb{E}}_s\left[e\left(n\right){\alpha }_{n}v\left(n+1\right)\right]=0$, so the numerator of Equation (15) can be simplified as

$${\mathbb{E}}_s\left[e\left(n\right){\alpha }_{n}r\left(n+1\right)\right]={\mathbb{E}}_s[e\left(n\right){\alpha }_n{\tilde{\mathit{w}}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n+1\right)].$$

(19)

Furthermore, approximate $\tilde{\mathit{w}}\left(n\right)$ as the projection of the input $\mathit{x}\left(n\right)$, that is $\tilde{\mathit{w}}\left(n\right)=\gamma \left(n\right)\mathit{x}\left(n\right)$, which yields

$$e\left(n\right)=d\left(n\right)-y\left(n\right)={\tilde{\mathit{w}}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n\right)=\gamma \left(n\right)\parallel \mathit{x}\left(n\right){\parallel }^2.$$

(20)

Thus, the instantaneous estimate of $\gamma \left(n\right)$ becomes $\gamma \left(n\right)={\displaystyle \frac{e\left(n\right)}{\parallel \mathit{x}\left(n\right){\parallel }^2}}$, which yields

$$\tilde{\mathit{w}}\left(n\right)={\displaystyle \frac{e\left(n\right)\mathit{x}\left(n\right)}{\parallel \mathit{x}\left(n\right){\parallel }^2}}.$$

(21)

If the input signal is assumed to be stationary, the numerator and denominator are approximately canceled, and the step size tends to a constant value (close to 1). However, in non-stationary environments, this approximation is clearly idealized, because the true projection of the coefficient error onto the input subspace is not always strictly equal to the single-sample estimate ${\displaystyle \frac{e\left(n\right)\mathit{x}\left(n\right)}{\parallel \mathit{x}\left(n\right){\parallel }^2}}$. A more accurate estimate should come from short-term least-squares or generalized linear estimation [32], which integrates information from several past samples and therefore includes correction factors related to the covariance matrix as well as residual terms. To characterize the signal dynamics, it is necessary to further perform a subspace decomposition of $\tilde{\mathit{w}}\left(n\right)$.

If two vectors lie in this subspace, then any linear combination of them also lies in the subspace. Considering that a single-sample estimate cannot capture covariance information [32], a smoothing correction is introduced:

$$\tilde{\mathit{w}}\left(n\right)=A\left(n\right){\displaystyle \frac{e\left(n\right)\mathit{x}\left(n\right)}{\parallel \mathit{x}\left(n\right){\parallel }^2}}+B\left(n\right){\displaystyle \frac{\mathsf{\zeta }\left(n\right)}{\parallel \mathit{x}\left(n\right){\parallel }^2}}$$

(22)

where $\mathsf{\zeta }\left(n\right)$ denotes the residual subspace component that cannot be described solely by $\mathit{x}\left(n\right)$ and $e\left(n\right)$. This step introduces an approximate projection of the coefficient error vector onto the input signal subspace. The subsequent decomposition further accounts for residual components that cannot be captured by single-sample estimation.

Substituting Equation (22) into Equation (10) (neglecting noise) yields

$$r\left(n+1\right)\approx A\left(n\right){\displaystyle \frac{e\left(n\right)\mathit{x}\left(n\right)\mathit{x}\left(n+1\right)}{\parallel \mathit{x}\left(n\right){\parallel }^2}}+B\left(n\right){\displaystyle \frac{{\mathsf{\zeta }}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n+1\right)}{\parallel \mathit{x}\left(n\right){\parallel }^2}}.$$

(23)

At this point, the numerator of Equation (15) becomes

$$\begin{array}{c}{\mathbb{E}}_s\left[e\left(n\right){\alpha }_{n}r\left(n+1\right)\right]\\ ={\mathbb{E}}_s\left[e\left(n\right){\alpha }_{n}A\left(n\right){\displaystyle \frac{e\left(n\right)\mathit{x}\left(n\right)\mathit{x}\left(n+1\right)}{\parallel \mathit{x}\left(n\right){\parallel }^2}}\right]+{\mathbb{E}}_s\left[e\left(n\right){\alpha }_{n}B\left(n\right){\displaystyle \frac{{\mathsf{\zeta }}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n+1\right)}{\parallel \mathit{x}\left(n\right){\parallel }^2}}\right].\end{array}$$

(24)

Approximating $\mathit{x}(n+1)$ by $\mathit{x}\left(n\right)$ as $\mathit{x}\left(n+1\right)=p\mathit{x}\left(n\right)+q$ (which holds within the short-time analysis window of speech or stationary inputs, and although high-transient regions may introduce errors, these will be suppressed by the subsequent smoothing process). Substituting this relation into Equation (24) reveals that the numerator and denominator contain terms of the form ${\mathbb{E}}_s\left[e^2\right(n)\hspace{0.17em}(\cdots \left)\right]$, which involve fourth-order or higher-order moments such as second- and fourth-order terms in $\parallel \mathit{x}\left(n\right){\parallel }^2$. To simplify the high-order moments involved, a weak Gaussian approximation is adopted, allowing fourth-order terms to be expressed in terms of second-order moments via Isserlis’ theorem [33]. This approach relies on the symmetry of even-order moments for Gaussian variables and the key assumption that the involved variables (or their linear combinations) can be approximated as zero-mean Gaussian variables with symmetric probability density functions. For any such variables $u_1,u_2,u_3,u_4$, this yields the decomposition:

$${\mathbb{E}}_s\left[u_1{u}_2{u}_3{u}_4\right]={\mathbb{E}}_s\left[u_1{u}_2\right]{\mathbb{E}}_s\left[u_3{u}_4\right]+{\mathbb{E}}_s\left[u_1{u}_3\right]{\mathbb{E}}_s\left[u_2{u}_4\right]+{\mathbb{E}}_s\left[u_1{u}_4\right]{\mathbb{E}}_s\left[u_2{u}_3\right].$$

(25)

In this way, the symmetry of the underlying distributions provides a tractable mechanism to reduce higher-order statistical dependencies into lower-order terms, which facilitates the derivation of an analytically interpretable step-size expression. Without this symmetry-related simplification, the analytical derivation would involve intractable higher-order moment terms and would be difficult to express in a closed form. It should be noted that this symmetry-based simplification is approximate and mainly holds under locally stationary conditions. Applying this decomposition idea to the fourth-order mixed terms in the numerator and denominator of Equation (15) yields

$${\mathbb{E}}_s\left[e\left(n\right){\alpha }_{n}r\left(n+1\right)\right]\approx c_1{\mathbb{E}}_s\left[e^2\left(n\right)\parallel \mathit{x}\left(n\right){\parallel }^2\right]+c_2{\mathbb{E}}_s\left[e^2\left(n\right)\right]+\cdots$$

(26)

$$\begin{array}{c}{\mathbb{E}}_s\left[{e\left(n\right)}^2{{\alpha }_n}^2\right]\\ \approx d_1{\mathbb{E}}_s\left[e^2\left(n\right){(\parallel \mathit{x}(n\left){\parallel }^2\right)}^2\right]+d_2{\mathbb{E}}_s\left[e^2\left(n\right)\parallel \mathit{x}\left(n\right){\parallel }^2\right]+d_3{\mathbb{E}}_s\left[e^2\left(n\right)\right]+\cdots \end{array}$$

(27)

where $c_i$ and $d_i$ are slowly varying coefficients.

When the input energy $\parallel \mathit{x}\left(n\right){\parallel }^2$ exhibits a significant dynamic range (that is, its order is higher than the constant terms), the dominant denominator term becomes $d_1{\mathbb{E}}_s\left[e^2\right(n)\parallel \mathit{x}(n\left){\parallel }^4\right]$ because ${\alpha }_n$ brings higher-order input energy, while the dominant numerator term becomes $c_1{\mathbb{E}}_s\left[e^2\right(n)\parallel \mathit{x}(n\left){\parallel }^2\right]$. In this case, taking the ratio of dominant terms yields

$$\mu \left(n\right)\approx {\displaystyle \frac{c_1{\mathbb{E}}_s\left[{e\left(n\right)}^2\parallel \mathit{x}\left(n\right){\parallel }^2\right]}{d_1{\mathbb{E}}_s\left[{e\left(n\right)}^2{\left(\parallel \mathit{x}\left(n\right){\parallel }^2\right)}^2\right]}}={\displaystyle \frac{c_1}{d_1}}\cdot {\displaystyle \frac{1}{{\mathbb{E}}_s\left[\parallel \mathit{x}\left(n\right){\parallel }^2\right]}}.$$

(28)

In this step, only the dominant terms are retained, while higher-order and less significant components are neglected, which leads to a simplified yet interpretable form.

In engineering implementations, the short-time averages are often replaced by instantaneous estimates, which yields the empirical instantaneous-driven form (scaled and regularized):

$$\mu \left(n\right)\propto {\displaystyle \frac{e^2\left(n\right)}{\parallel \mathit{x}\left(n\right){\parallel }^2}}.$$

(29)

Directly using $\mu (n)={\displaystyle \frac{e^2\left(n\right)}{\parallel \mathit{x}\left(n\right){\parallel }^2}}$ would cause the step size to become excessively large when the input energy is low and may even lead to numerical divergence. To improve robustness, this form is modified as follows. The denominator is regularized by replacing $\parallel \mathit{x}\left(n\right){\parallel }^2$ with ${\parallel \mathit{x}\left(n\right)\parallel }^2+\epsilon$ to avoid division by zero. A joint normalization term ${\displaystyle \frac{e^2\left(n\right)}{{\parallel \mathit{x}\left(n\right)\parallel }^2+e^2\left(n\right)+\epsilon }}$ is applied to balance the contributions of the error and the input. The square root of the result is taken to smooth the dynamic range and stabilize the step-size variation. This yields the practical reference step size:

$${\mu }_{ref}\left(n\right)=\sqrt{{\displaystyle \frac{e^2\left(n\right)}{{\parallel \mathit{x}(n)\parallel }^2+e^2\left(n\right)+\epsilon }}}.$$

(30)

When the error energy $e^2\left(n\right)$ is large, and the input energy is small, ${\mu }_{ref}(n)$ approaches 1, which tends to promote faster convergence. When the error energy becomes stable or the input energy increases, ${\mu }_{ref}(n)$ is compressed, which helps maintain steady-state performance.

It should be noted that the above derivation relies on standard simplifying assumptions commonly adopted in adaptive filtering analysis. As a result, the obtained step-size expression (Equation (30)) is not claimed to be globally optimal in a strict theoretical sense. Instead, it serves as a reference form that captures the general dependence of the step size on the error energy and the input signal power.

In particular, the derived step size can be regarded as a reasonable reference choice under the considered assumptions:

• The input signal statistics are approximately stationary or slowly time-varying within a short time window;
• The error signal can be locally approximated as a zero-mean Gaussian process;
• The weight perturbations between successive iterations are sufficiently small.

From a practical perspective, the analytically derived ${\mu }_{ref}\left(n\right)$ can therefore be interpreted as an approximate solution obtained by retaining the dominant terms in the instantaneous MSE recursion under locally stationary assumptions. It provides a structured reference that reflects the desired step-size trend while remaining sensitive to instantaneous estimation noise and modeling mismatch.

It should be emphasized that ${\mu }_{ref}\left(n\right)$ is not intended to be the final step size used in the adaptive update. When used alone, it reflects the desired dependency of the step size on the error energy and the input signal power but may suffer from performance degradation under strong non-stationarity and noisy instantaneous estimates. Nevertheless, ${\mu }_{ref}\left(n\right)$ is bounded within the interval $\left(\mathrm{0,1}\right]$, which is consistent with the stability condition of the NLMS algorithm.

Outside these conditions, the analytical expression alone may not provide sufficient robustness, which motivates the use of a learning-based predictor in the proposed framework.

2.3. NN Prediction, Dynamic Modulation and Steady-State Constraint Mechanism

Theoretically, Equation (30) can adjust the step size automatically according to the magnitude of the error. In practical environments, the expected performance is not always achieved. This is mainly because the expression in Equation (30) is derived under specific signal models and instantaneous statistical conditions, and its effectiveness depends on assumptions on the input distribution and error characteristics. As a result, the formula shows limited generalization capability, and changes in signal conditions, especially under strongly non-stationary scenarios, may lead to abnormal step-size behavior.

To address this issue, this paper introduces an NN as an auxiliary tool for step-size adjustment, aiming to improve the adaptability of the analytically derived step-size under practical conditions. The NN takes an eight-dimensional feature vector as input. These features include the input signal power, error magnitude and energy, normalized error terms, and energy ratio-based quantities. They capture the instantaneous deviation, local energy, and short-term statistical behavior of the filter, and describe the filter state and signal characteristics at different convergence stages:

$$\mathit{f}\left(n\right)=\left[{\displaystyle \frac{x_2\left(n\right)}{L}},\left|e\left(n\right)\right|,e^2\left(n\right),\overline{e^2}\left(n\right),{\displaystyle \frac{\overline{x_2}\left(n\right)}{L}},{\displaystyle \frac{e\left(n\right)}{\sqrt{x_2\left(n\right)}+\epsilon }},\overline{{\varphi }_1}\left(n\right),\overline{{\varphi }_2}\left(n\right)\right]$$

(31)

where ${\displaystyle \frac{x_2\left(n\right)}{L}}={\displaystyle \frac{\parallel \mathit{x}\left(n\right){\parallel }^2}{L}}$ denotes the average input signal power, and the normalization by $L$ ensures scale consistent with respect to the filter length. $\left|e\left(n\right)\right|$ and $e^2\left(n\right)$ characterize the error magnitude and energy; $\overline{e^2}\left(n\right)$ and $\overline{x_2}\left(n\right)$ is the smoothed estimate of $e^2(n)$ and $x_2\left(n\right)$; ${\displaystyle \frac{e\left(n\right)}{\sqrt{x_2\left(n\right)}+\epsilon }}$ provides a normalized signed error, which reflects both the magnitude and direction of the error relative to the input; $\overline{{\varphi }_1}\left(n\right)$ and $\overline{{\varphi }_2}\left(n\right)$ are the smoothed estimates of ${\varphi }_1(n)={\displaystyle \frac{e^2\left(n\right)}{x_2\left(n\right)+\epsilon }}$ and ${\varphi }_2(n)={\displaystyle \frac{e^2\left(n\right)}{e^2\left(n\right)+x_2\left(n\right)+\epsilon }}$, capturing the relative contribution of the error energy with respect to the input and total energy, respectively; $\epsilon$ is a small constant preventing division by zero.

This work employs a two-layer fully connected NN. The input layer size matches the feature dimension; the two hidden layers contain 64 and 24 neurons, respectively. ReLU activation [34] is used to enhance nonlinear mapping capability, and light dropout is applied to reduce overfitting. The output layer uses a Sigmoid function to constrain the predicted step size to the interval $\left(\mathrm{0,1}\right)$, which helps constrain the output within a bounded range. During the training phase, the analytically derived reference step size ${\mu }_{ref}\left(n\right)$ is directly used as the supervision signal. In this sense, ${\mu }_{ref}\left(n\right)$ can be viewed as a practical surrogate target, while the NN aims to learn a smoother and more robust mapping from signal features to step-size adjustment under real signal fluctuations. However, it should be emphasized that the ${\mu }_{ref}\left(n\right)$ is computed from instantaneous signal observations under practical conditions, and therefore inherently contains statistical fluctuations and modeling inaccuracies. From this perspective, the NN is not intended to replicate the analytical expression itself, but to learn the underlying nonlinear relationship between signal features and the desired step-size behavior in practical scenarios. The discrepancies between the analytical model and its practical realization arise from various simplifying assumptions, such as local stationarity, projection approximations, and Gaussian moment approximations, which are difficult to model explicitly in closed form. As a result, directly refining the reference step size using analytical methods would require complex high-order statistical modeling, which is generally difficult to implement in real-time adaptive filtering. The NN provides a practical alternative by learning this relationship from data through its nonlinear mapping capability. In this sense, the NN is expected to capture the discrepancy between the idealized analytical model and its practical realization, which can be interpreted as a form of data-driven correction to the analytically motivated step-size structure. In the inference stage, the step size predicted by the NN is further processed through dynamic modulation, constraint, and smoothing operations to enhance stability and temporal consistency. These mechanisms operate on top of the NN output and are designed to regulate its temporal behavior, whereas the NN itself focuses on capturing the instantaneous nonlinear mapping between signal features and step-size adjustment. Based on this learning objective, the practical training process of the network is described as follows. Training samples with different SNRs are constructed, and the network parameters are updated by minimizing the loss between the NN-predicted step-size ${\mu }_{pred}\left(n\right)$ and ${\mu }_{ref}\left(n\right)$, enabling the NN to learn the step-size adjustment patterns under different noise conditions and different convergence stages. During the inference phase, the network generates the predicted step size ${\mu }_{pred}\left(n\right)$ based on the instantaneous signal features. This data-driven correction refines the reference step size and forms a nonlinear step-size adjustment mechanism that is expected to improve robustness and generalization under varying conditions.

The network is trained using the Adam optimizer with an initial learning rate of ${10}^{-3}$. The mini-batch size is set to 64 and the training process runs for 40 epochs. To improve generalization ability, a dropout rate of 0.1 is applied in the hidden layers and an $L_2$ regularization term with coefficient ${10}^{-4}$ is adopted. Early stopping is employed based on the validation loss with a patience of six validation checks to prevent overfitting.

However, the NN generates the step size mainly based on instantaneous signal features, focusing on the mapping at the current time instant rather than explicitly reflecting the temporal evolution of the step size. In rapidly time-varying scenarios, the predicted step size may exhibit instantaneous fluctuations. Without further regulation, this behavior may lead to excessively aggressive filter updates or degraded steady-state performance. Therefore, on top of the NN-predicted step size, this work introduces a dynamic modulation and stabilization mechanism to enhance the robustness of the algorithm.

Specifically, an exponential-decay amplification factor is employed as a dynamic modulation term to accelerate early-stage convergence and gradually restore stability. The amplification factor is computed as follows:

$$boostFactor(n)=\gamma \cdot e^{-{\displaystyle \frac{\beta \cdot n}{N}}}$$

(32)

where $\gamma$ is the initial amplification coefficient, $\beta$ is a hyperparameter controlling the decay rate, $n$ is the current iteration index, and $N$ is the total number of iterations. At this time, the step size update is defined as

$${\mu }_{boost}\left(n\right)={\mu }_{pred}(n)\cdot \left(1+boostFactor\left(n\right)\right)$$

(33)

where ${\mu }_{pred}(n)$ denotes the step-size predicted by the NN.

The step-size is limited to the interval $[{\mu }_{min},{\mu }_{max}]$ to avoid instability caused by excessively large or small values, which leads to

$${\mu }_{constr}\left(n\right)=\mathit{min}\left(\mathit{max}\left({\mu }_{boost}\left(n\right),{\mu }_{min}\right),{\mu }_{max}\right)$$

(34)

where ${\mu }_{max}$ is the maximum step-size and ${\mu }_{min}$ is the minimum step-size.

Furthermore, the parameter $maxStepRatio$ is used to restrict the change in the step size with respect to its previous value, yielding

$${\mu }_{ratio}(n)=\mathit{min}\left({\mu }_{constr}\left(n\right),{\mu }_{prev}\cdot maxStepRatio\right)$$

(35)

$${\mu }_{ratio}(n)=\mathit{max}\left({\mu }_{ratio}(n),{\displaystyle \frac{{\mu }_{prev}}{maxStepRatio}}\right)$$

(36)

where ${\mu }_{prev}$ denotes the step-size at the previous time instant.

A smoothing operation is then applied to obtain the final step-size used in the filter update:

$${\mu }_{final}\left(n\right)=\alpha {\mu }_{ratio}\left(n\right)+\left(1-\alpha \right){\mu }_{prev}$$

(37)

where $\alpha \in \left(\mathrm{0,1}\right)$ is the smoothing coefficient.

Through these operations, the step size predicted by the NN is converted into a step size used in the filter update. The flowchart of the NN-VSS-NLMS algorithm is shown in Figure 2.

2.4. Convergence Analysis

For the conventional NLMS algorithm, mean-square stability is guaranteed when the step size satisfies $0<\mu <2$.

In the proposed method, the reference step size is constructed within a predefined interval $[{\mu }_{min},{\mu }_{max}]$, where $0<{\mu }_{min}<{\mu }_{max}<2$. Through subsequent modulation and constraint operations, the step size used in the coefficient update remains within this interval.

Therefore, the effective step size involved in the adaptive update is consistent with the stability range typically associated with the NLMS algorithm, which supports stable operation under general conditions.

To further analyze the behavior of the proposed NN-VSS-NLMS algorithm under non-stationary conditions, the evolution of the coefficient-error vector $\tilde{\mathit{w}}\left(n\right)$ is considered. The update equation is given by

$$\tilde{\mathit{w}}\left(n+1\right)=\tilde{\mathit{w}}\left(n\right)-\mu \left(n\right){\displaystyle \frac{e\left(n\right)\mathit{x}\left(n\right)}{{\parallel \mathit{x}\left(n\right)\parallel }^2+\delta }}.$$

(38)

Taking the expectation of the norm squared of Equation (38) yields

$$\begin{array}{c}E\left[\parallel \tilde{\mathit{w}}\left(n+1\right){\parallel }^2\right]\\ =E\left[\parallel \tilde{\mathit{w}}\left(n\right){\parallel }^2\right]-2E\left[\mu \left(n\right){\displaystyle \frac{e\left(n\right){\tilde{\mathit{w}}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n\right)}{{\parallel \mathit{x}\left(n\right)\parallel }^2+\delta }}\right]+E\left[{\mu }^2\left(n\right){\displaystyle \frac{e^2\left(n\right)\parallel \mathit{x}\left(n\right){\parallel }^2}{({\parallel \mathit{x}\left(n\right)\parallel }^2+\delta {)}^2}}\right].\end{array}$$

(39)

Since error signal $e(n)={\tilde{\mathit{w}}}^{\mathrm{T}}(n)\mathit{x}(n)+v(n)$, where the noise $v\left(n\right)$ is uncorrelated with both $\tilde{\mathit{w}}\left(n\right)$ and $\mathit{x}\left(n\right)$, it follows that

$$E[e\left(n\right){\tilde{\mathit{w}}}^{\mathrm{T}}\left(n\right)\mathit{x}\left(n\right)]=E[\parallel {\tilde{\mathit{w}}}^{\mathrm{T}}(n)\mathit{x}(n){\parallel }^2]\ge 0.$$

(40)

Under commonly adopted assumptions in adaptive filtering, the first-order term in Equation (39) generally dominates, while the higher-order term associated with ${\mu }^2\left(n\right)$ has a relatively smaller influence. In this case, a strict theoretical proof of monotonic convergence is difficult to establish, while the above analysis still reflects the overall convergence behavior.

However, it should be noted that the step size $\mu \left(n\right)$ is jointly determined by the analytical model and the data-driven prediction, and thus varies with the instantaneous signal conditions. In this case, a strict theoretical proof of monotonic convergence is difficult to establish, while the above analysis still reflects the overall convergence behavior.

2.5. Computational Complexity Analysis

This subsection analyzes the computational complexity of the considered algorithms in terms of the number of operations per iteration. Specifically, the total computational cost is divided into two components: operations proportional to the filter length $L$ and constant operations independent of $L$.

For the NLMS algorithm, the dominant computations per iteration include the output calculation, input energy estimation, and weight update. These operations result in approximately $3L+1$ arithmetic operations, leading to an overall computational complexity of O($L$).

The NP-VSS-NLMS and VSS-NLMS algorithms introduce additional recursive estimations and step-size adaptation mechanisms. These operations incur a limited number of extra scalar computations, while preserving the overall O($L$) complexity.

For the proposed NN-VSS-NLMS algorithm, the adaptive filtering update follows the same structure as NLMS, and thus retains the O($L$) complexity with respect to $L$. The additional computational cost arises from feature extraction, NN inference, and step-size control. The feature extraction process involves only a small number of arithmetic operations (approximately 14 operations) and remains independent of $L$. The step-size control includes dynamic boosting, bounding constraints, and temporal smoothing. These operations mainly consist of multiplications, comparisons, and exponential evaluations, introducing only a small constant computational overhead (approximately 10 operations). The computational cost of the NN depends on its architecture. For the adopted 8–64–24–1 fully connected structure, the forward-pass computation requires approximately $8\times 64+64\times 24+24\times 1\approx 2100$ arithmetic operations per iteration. Therefore, the additional complexity of the proposed algorithm is primarily dominated by the NN inference, while the feature extraction and step-size control introduce only minor constant overhead. Consequently, the overall computational complexity remains O($L$).

A summary of the computational complexity of the considered algorithms is provided in

Table 1. Computational complexity comparison of the considered algorithms.

Algorithm	Order	Operations Dependent on $\mathit{L}$	Fixed Overhead
NLMS	$\mathrm{O}\left(L\right)$	$3L+1$	1
NP-VSS-NLMS	$\mathrm{O}\left(L\right)$	$3L+1$	$\approx$10
VSS-NLMS	$\mathrm{O}\left(L\right)$	$4L+1$	$\approx$20
NN-VSS-NLMS	$\mathrm{O}\left(L\right)$	$3L+1$	$\approx$2100

.

3. Algorithm Simulation and Analysis

To validate the performance of the proposed NN-VSS-NLMS algorithm, this paper conducts comprehensive simulations on the MATLAB R2024a platform under both stationary and non-stationary input conditions.

For a meaningful and balanced comparison, several representative algorithms are selected as benchmarks. The NP-VSS-NLMS algorithm adjusts the step size according to the relationship between error power and noise power, resulting in a direct and responsive adaptation mechanism. The VSS-NLMS algorithm proposed by Huang et al. employs recursive estimation of signal and noise statistics, leading to a more structured step-size adaptation process. The NLMS algorithm is included as a baseline due to its simplicity and well-established performance characteristics. These algorithms represent different approaches to step-size adaptation and together provide a comprehensive basis for performance evaluation. In contrast, the proposed NN-VSS-NLMS algorithm integrates an analytically derived reference step size with NN-assisted refinement, providing a more flexible and controllable step-size adaptation mechanism.

3.1. Input Signals

Two representative signals are used for the experiments in stationary environments. The first one is white noise that follows a standard normal distribution with zero mean and unit variance [35], which is employed to evaluate the algorithm’s performance under uncorrelated inputs. The second one is an AR(1) signal generated by $x\left(n\right)=ax(n-1)+v\left(n\right)$, where $a$ is the autoregressive coefficient and $v\left(n\right)$ is white noise. This signal is used to assess the algorithm’s behavior when dealing with colored inputs.

For the experiments in non-stationary environments, speech signals are selected from the LibriSpeech corpus. Specifically, a total of 1000 speech recordings from different speakers are used for training, while 100 recordings from another set of speakers are used for testing to ensure speaker independence between the training and test sets. All recordings are resampled to 8 kHz. To ensure consistency in the adaptive filtering process, each speech signal is truncated to a fixed length of $N=2000$ samples. This fixed-length segmentation ensures uniform input conditions and facilitates stable feature extraction and training. Speech signals inherently exhibit strong non-stationary characteristics and temporal variations, making them suitable for evaluating the algorithm’s performance under realistic and time-varying conditions.

3.2. Algorithm Parameter Configuration

The parameter settings of all algorithms are summarized in

Table 2. Simulation parameters of our NN-VSS-NLMS and other comparing algorithms.

Algorithm:	NLMS
Parameters:	${\mu }_{NLMS}=0.01$
Algorithm:	Benesty’s NP-VSS-NLMS [16] $\mu =\left\{\begin{array}{c}1-{\displaystyle \frac{{\widehat{\sigma }}_v}{\zeta +{\widehat{\sigma }}_e\left(n\right)}}, \mathrm{if} {\widehat{\sigma }}_e\left(n\right)\ge {\widehat{\sigma }}_v\\ 0, otherwise\end{array}\right.$ ${{\widehat{\sigma }}_e}^2\left(n\right)=\kappa {{\widehat{\sigma }}_e}^2\left(n-1\right)+(1-\kappa )e^2(n)$
Parameters:	$\zeta ={\displaystyle \frac{{\widehat{\sigma }}_v}{1000}} , \delta ={10}^{-3} , \kappa =0.95$
Algorithm:	Huang’s VSS-NLMS [17] $\mu =\left\{\begin{array}{c}\alpha \mu \left(n-1\right)+\left(1-\alpha \right){\displaystyle \frac{{{\widehat{\sigma }}_e}^2\left(n\right)}{\beta {{\widehat{\sigma }}_v}^2\left(n\right)}}, \mathrm{if} \zeta \left(n\right)<{\mathsf{\zeta }}_{th}\\ 1, otherwise\end{array}\right.$
Parameters:	$\alpha =0.998 , \beta =30 , \epsilon =0.1 , {\mu }_{min}={10}^{-5} , {\mu }_{max}=1 , \mu \left(0\right)=1$
Algorithm:	Proposed NN-VSS-NLMS ${\mu }_{boost}(n)={\mu }_{pred}(n)\cdot (1+\gamma \cdot e^{-{\displaystyle \frac{\beta \cdot n}{N}}})$ ${\mu }_{constr}\left(n\right)=min(max({\mu }_{boost}(n),{\mu }_{min}),{\mu }_{max})$ ${\mu }_{ratio}(n)=max(min({\mu }_{constr}\left(n\right),{\mu }_{prev}\cdot maxStepRatio),{\displaystyle \frac{{\mu }_{prev}}{maxStepRatio}})$ ${\mu }_{final}\left(n\right)=\alpha {\mu }_{ratio}(n)+(1-\alpha ){\mu }_{prev}$
Parameters:	$\gamma =6.8 , \beta =12 , {\mu }_{min}={10}^{-4} , {\mu }_{max}=1 , maxStepRatio=2 , \alpha =0.9$

. The parameters of the compared algorithms are selected according to the recommended settings reported in their original publications and are used directly in the experiments without further retuning in order to ensure a fair and unbiased comparison. The hyperparameters in the proposed algorithm are determined through a combination of theoretical analysis and empirical tuning. Specifically, the step-size bounds (${\mu }_{min}$ and ${\mu }_{max}$) are selected according to the stability conditions of NLMS to ensure convergence. The remaining parameters (e.g., $\gamma$, $\beta$, $maxStepRatio$, and $\alpha$) are tuned through repeated experiments under different signal types (white noise, AR(1), and speech) and various SNR conditions. The final parameter configuration is chosen to achieve a favorable trade-off between convergence rate and steady-state error. In the experiments, the order of the adaptive filter is set to $L=8$ and $L=32$ for separate tests, and the sample length per iteration is $N=2000$. This allows the evaluation of the proposed algorithm under both low-order and higher-order filter conditions.

During the NN training stage, the proposed algorithm uses the Adam optimizer to update the network parameters, with an initial learning rate of 0.001. The SNR of the training data is randomly selected from 0, 5, 10, 15, and 20 dB, enabling the network to learn the step-size adaptation behavior under different noise conditions. The target labels for supervised learning are generated based on the reference step size derived in the previous section. The constructed dataset is randomly shuffled and divided into training and validation subsets. Approximately 10% of the samples are used for validation. This split ensures sufficient training data while enabling monitoring of generalization performance.

In the inference stage, all algorithm comparisons are conducted under a fixed SNR of 20 dB. The test set contains 100 speech samples from 30 different speakers to ensure diversity and independence. For each experimental setting, the reported results are obtained by averaging over 100 independent trials. Specifically, the MSE curves are first computed for each test sample and then averaged across all samples to produce the final performance curves.

3.3. Performance Metrics

This study adopts the MSE as the primary performance metric, based on which both the convergence rate and the steady-state error are evaluated. The MSE is defined as

$$MSE=10{\mathrm{l}\mathrm{o}\mathrm{g}}_{10}\left(E\right\{|{e\left(n\right)|}^2\}).$$

(41)

The convergence rate is defined as the average decrease in the MSE per iteration (unit: dB/iteration), computed as

$$R={\displaystyle \frac{{MSE}_{start}-{MSE}_{stable}}{N_{conv}}}$$

(42)

where ${MSE}_{start}$ denotes the average MSE over the first 50 iterations, ${MSE}_{stable}$ denotes the average MSE during the steady-state phase, and $N_{conv}$ is the number of iterations from the beginning until the algorithm enters steady state.

The steady-state error is defined as the average value of the MSE after the MSE curve enters the steady-state region. The steady-state region is determined as follows: when the fluctuation amplitude of the MSE curve, estimated using a sliding standard deviation with a window length of 50, falls below a threshold and remains below this value for at least 200 iterations, the algorithm is considered to have entered steady state. If this condition is not satisfied throughout the entire process, the last 20% of the iterations are regarded as the steady-state interval. The threshold is set according to the type of input signal, where 0.3 is used for stationary inputs and 1.0 is used for non-stationary inputs.

To further evaluate the statistical stability of the algorithms, the standard deviation of the steady-state MSE is also considered. For each algorithm, multiple independent trials are conducted under the same experimental settings, resulting in a set of steady-state MSE values.

Specifically, let ${MSE}_k$ denote the steady-state MSE obtained from the $k$-th trial. The mean and standard deviation are computed as

$${\mu }_{MSE}={\displaystyle \frac{1}{K}}\sum _{k=1}^K{MSE}_k$$

(43)

$${\sigma }_{MSE}=\sqrt{{\displaystyle \frac{1}{K}}\sum _{k=1}^K{\left({MSE}_k−{\mu }_{MSE}\right)}^2}$$

(44)

where $K$ is the number of trials. The MSE values are averaged in the linear domain and then converted to the dB scale for reporting.

The standard deviation reflects the variability of the algorithm performance across different trials. A smaller value indicates more stable and consistent behavior.

To evaluate the tracking ability of the algorithm under time-varying systems, the true FIR filter is multiplied by a triangular gain function (with amplitude varying from 1 to 3.5) within the iteration interval $\left[\mathrm{600,800}\right]$. This constructs a time-varying system for testing the algorithm under non-stationary conditions (as shown in Figure 3).

3.4. Performance Analysis of Step-Size Forms

To clarify the individual roles of different step-size components in the proposed NN-VSS-NLMS algorithm, a structured ablation study is conducted with the adaptive filter order set to $L=32$. Seven step-size variants are evaluated under identical conditions, including the reference step size ${\mu }_{ref}(n)$, the NN prediction ${\mu }_{pred}(n)$, the boosted step size ${\mu }_{boost}(n)$, the amplitude-constrained step size ${\mu }_{constr}\left(n\right)$, the ratio-constrained step size ${\mu }_{ratio}(n)$, the final smoothed step size ${\mu }_{final}\left(n\right)$, and the step size ${\mu }_{noNN}(n)$ obtained by applying boosting, constraint, and smoothing directly to the ${\mu }_{ref}(n)$ without introducing the NN. For white noise input, the MSE curves and step-size curves are shown in Figure 4 and Figure 5, respectively. For speech input, the MSE curves and step-size curves are shown in Figure 6 and Figure 7, respectively.

The ${\mu }_{ref}(n)$ provides a baseline evolution with relatively small magnitude, leading to a more conservative adaptation process. However, under non-stationary speech conditions, ${\mu }_{ref}(n)$ exhibits noticeable fluctuations in the steady-state phase, suggesting limited robustness to rapidly varying signal statistics.

With the introduction of NN, ${\mu }_{pred}(n)$ exhibits larger values, leading to a more responsive adaptation process. Compared with ${\mu }_{ref}(n)$, it achieves a faster reduction in MSE after the initial stage.

It should be noted that ${\mu }_{boost}(n)$ is introduced as an intermediate modulation signal in the step-size generation process, aiming to enhance the NN output during the early adaptation stage. Since this step-size form does not yet incorporate sufficient stability-oriented constraints for NLMS adaptation, it does not constitute an independent step-size strategy, and its corresponding MSE performance is therefore not taken as a primary evaluation metric.

After introducing constraints, ${\mu }_{constr}\left(n\right)$ and ${\mu }_{ratio}(n)$ exhibit very similar step-size trajectories, indicating that they produce comparable step-size ranges. Their corresponding MSE curves almost overlap, suggesting that these mechanisms primarily regulate the step-size behavior rather than directly reducing the error level.

Finally, the ${\mu }_{final}\left(n\right)$ achieves favorable performance under both stationary and non-stationary inputs, exhibiting a fast convergence rate and a low steady-state error. These results indicate that the analytically motivated reference step size alone is insufficient under non-stationary conditions, and that the learning-assisted correction and subsequent regulation are necessary to maintain stable and robust adaptation.

It is worth noting that the NN and the subsequent heuristic mechanisms play different roles in the proposed framework. The NN focuses on learning the instantaneous nonlinear mapping between signal features and step-size adjustment, while the dynamic modulation and constraint mechanisms regulate the temporal evolution and stability of the step size. It can be observed from the figures that ${\mu }_{final}\left(n\right)$ exhibits superior overall performance to ${\mu }_{noNN}(n)$. Therefore, the improvement is mainly attributed to the enhanced step-size adaptation capability introduced by the NN, which enables a more accurate characterization of the desired step-size behavior under non-stationary conditions. The subsequent heuristic mechanisms further refine this adaptation by improving stability and preventing undesirable fluctuations.

3.5. Performance Analysis Under Stationary Inputs

Figure 8 and Figure 9 presents the performance of the proposed NN-VSS-NLMS algorithm compared with NLMS, NP-VSS-NLMS, and VSS-NLMS under white-noise input, for filter orders of 8 and 32 respectively. The corresponding quantitative results are provided in

Table 3. Performance metrics of different algorithms with white noise input in a time-invariant system at SNR = 20 dB.

$\mathbf{Filter} \mathbf{Order} \mathit{L}$	Metric	NLMS	NP-VSS-NLMS	VSS-NLMS	NN-VSS-NLMS
8	Convergence Rate (dB/iteration)	0.010839	0.053242	0.049901	0.054135
	Steady-state Error (dB)	−15.495	−23.655	−23.426	−23.752
	Standard Deviations (dB)	0.40742	0.41015	0.34955	0.35424
32	Convergence Rate (dB/iteration)	\	0.044393	0.042922	0.044432
	Steady-state Error (dB)	\	−22.672	−22.587	−22.779
	Standard Deviations (dB)	0.30585	0.34954	0.33223	0.33409

.

For $L=8$, the proposed algorithm achieves a higher convergence rate than NP-VSS-NLMS and VSS-NLMS, indicating faster adaptation during the initial stage. In the steady-state phase, it attains a lower steady-state error than NLMS and remains slightly below NP-VSS-NLMS and VSS-NLMS, reflecting a moderate improvement in steady-state accuracy. In addition, its standard deviation is close to that of VSS-NLMS and lower than that of NP-VSS-NLMS, indicating that the steady-state performance is achieved without a noticeable increase in fluctuation level.

For $L=32$, the overall convergence rate decreases due to the increased system order, and the differences among the variable step-size algorithms become smaller. Under this condition, NN-VSS-NLMS still yields a faster convergence rate, although the margin over NP-VSS-NLMS becomes limited, while remaining slightly faster than VSS-NLMS. In the steady-state phase, it continues to produce a slightly lower steady-state error, and its standard deviation remains within a similar range to the other algorithms, indicating that the fluctuation level does not increase with filter order.

Figure 10 and Figure 11 show the performance of the proposed NN-VSS-NLMS algorithm and the NLMS, NP-VSS-NLMS, and VSS-NLMS algorithms under AR(1) inputs with four correlation levels ($a=0.3, 0.5, 0.7, 0.9$) for filter orders of 8 and 32, respectively. The corresponding quantitative results are provided in

Table 4. Performance metrics of different algorithms with AR(1) input in a time-invariant system at SNR = 20 dB for different correlation coefficients ($a=0.3, 0.5, 0.7, 0.9$).

Filter Order $\mathit{L}$	Correlation Level $\mathit{a}$	Metric	NLMS	NP-VSS-NLMS	VSS-NLMS	NN-VSS-NLMS
8	0.3	Convergence Rate (dB/iteration)	0.01245	0.062827	0.064369	0.068016
		Steady-state Error (dB)	−17.641	−21.758	−21.512	−21.955
		Standard Deviations (dB)	0.29186	0.34086	0.31145	0.31162
	0.5	Convergence Rate (dB/iteration)	0.01437	0.067417	0.057339	0.06983
		Steady-state Error (dB)	−15.642	−20.043	−19.811	−20.252
		Standard Deviations (dB)	0.3216	0.36726	0.33839	0.34103
	0.7	Convergence Rate (dB/iteration)	0.014548	0.054301	0.04367	0.057732
		Steady-state Error (dB)	−15.391	−17.682	−17.404	−17.873
		Standard Deviations (dB)	0.43746	0.41572	0.38412	0.38363
	0.9	Convergence Rate (dB/iteration)	0.019465	0.037923	0.012667	0.037146
		Steady-state Error (dB)	−10.086	−12.798	−12.687	−12.903
		Standard Deviations (dB)	0.51497	0.55338	0.52526	0.53198
32	0.3	Convergence Rate (dB/iteration)	\	0.053511	0.050999	0.054507
		Steady-state Error (dB)	\	−20.845	−20.746	−20.853
		Standard Deviations (dB)	0.37233	0.3827	0.34918	0.36255
	0.5	Convergence Rate (dB/iteration)	\	0.04374	0.043561	0.045154
		Steady-state Error (dB)	\	−19.461	−19.355	−19.449
		Standard Deviations (dB)	0.33843	0.34599	0.31988	0.32531
	0.7	Convergence Rate (dB/iteration)	\	0.029781	0.029003	0.031949
		Steady-state Error (dB)	\	−17.195	−17.161	−17.204
		Standard Deviations (dB)	0.40094	0.37122	0.36554	0.36818
	0.9	Convergence Rate (dB/iteration)	\	0.02369	0.021877	0.026133
		Steady-state Error (dB)	\	−12.236	−12.233	−12.256
		Standard Deviations (dB)	0.41295	0.48468	0.44401	0.45335

.

For $L=8$, NN-VSS-NLMS maintains a relatively higher convergence rate across most correlation levels. As the input correlation increases, the convergence rates of all algorithms decrease, and the differences among the variable step-size algorithms become smaller. In the steady-state phase, NN-VSS-NLMS consistently achieves a lower steady-state error than the other algorithms.

For $L=32$, a similar overall behavior is observed. NN-VSS-NLMS maintains a higher convergence rate than NP-VSS-NLMS and VSS-NLMS across different correlation levels. As the correlation increases, the margin in convergence rate gradually decreases. In the steady-state phase, the performance of the variable step-size algorithms becomes very close. NN-VSS-NLMS generally maintains a slightly lower steady-state error across different correlation levels. The standard deviations of the variable step-size algorithms remain within similar ranges under all conditions, indicating comparable fluctuation levels.

Overall, under both white noise and AR(1) stationary inputs, the proposed algorithm demonstrates better performance, validating its capability of adaptively adjusting the step-size in stationary environments.

3.6. Performance Analysis Under Non-Stationary Inputs

Non-stationary inputs represent typical conditions in real-world applications, where the algorithm must maintain convergence and tracking ability when the system or the input statistics undergo abrupt changes. Figure 12, Figure 13, Figure 14 and Figure 15 present the performance of the proposed NN-VSS-NLMS algorithm, NLMS, NP-VSS-NLMS, and VSS-NLMS under speech inputs for both time-invariant and time-varying systems with filter orders of 8 and 32. Specifically, Figure 12 and Figure 13 correspond to the time-invariant system with $L=8$ and $L=32$, respectively, while Figure 14 and Figure 15 illustrate the corresponding results for the time-varying system. The corresponding quantitative results for the time-invariant system are provided in

Table 5. Performance metrics of different algorithms with speech input in a time-invariant system at SNR = 20 dB.

$\mathbf{Filter} \mathbf{Order} \mathit{L}$	Metric	NLMS	NP-VSS-NLMS	VSS-NLMS	NN-VSS-NLMS
8	Convergence Rate (dB/iteration)	0.043339	0.049149	0.018884	0.057867
	Steady-state Error (dB)	−63.886	−67.766	−55.087	−69.673
	Standard Deviations (dB)	0.21326	0.21728	0.14934	0.32904
32	Convergence Rate (dB/iteration)	0.0086559	0.021447	0.0063509	0.027948
	Steady-state Error (dB)	−58.303	−66.491	−55.852	−68.682
	Standard Deviations (dB)	0.22064	0.26039	0.17306	0.25474

.

For the time-invariant system, the corresponding quantitative results are provided in Table 5. When $L=8$, the proposed algorithm exhibits a higher convergence rate (0.057867 dB/iteration) than the other algorithms, indicating a faster adaptation process under speech input. In the steady-state phase, it achieves a lower steady-state error (−69.673 dB), reflecting improved estimation accuracy. It is also observed that the standard deviation of NN-VSS-NLMS is relatively larger under this condition. This behavior can be associated with the more responsive step-size adjustment, which facilitates faster convergence while introducing slightly increased fluctuations around the steady state. However, the magnitude of this increase remains limited compared to the gain in steady-state accuracy.

When the filter order increases to $L=32$, the convergence rate decreases due to the higher system dimensionality, while NN-VSS-NLMS still maintains a relatively higher convergence rate (0.027948 dB/iteration) among the compared algorithms. In steady state, it continues to achieve a lower steady-state error (–68.682 dB). In this case, the standard deviation becomes closer to those of NP-VSS-NLMS and VSS-NLMS, indicating that the fluctuation level is moderated as the filter order increases.

In time-varying systems (i.e., non-stationary environments), the tracking performance of all algorithms is evaluated using a variable filter, whose evolution pattern is shown in Figure 3 [36]. The main challenge in this scenario is that the system changes occur during periods when the step-size tends to be small, while VSS-NLMS algorithms must increase the step-size promptly to respond to system variations [36]. For $\mathrm{L}=8$, it can be observed that the proposed NN-VSS-NLMS algorithm responds effectively to the system change, with the MSE increasing during the abrupt transition and then decreasing as adaptation resumes. Compared with NLMS, the proposed algorithm maintains a lower MSE level throughout the transition process. In comparison with NP-VSS-NLMS and VSS-NLMS, the overall MSE trajectory after the abrupt change remains at a relatively lower level, indicating improved tracking performance.

For $L=32$, a similar trend is observed, while the adaptation process becomes slower due to the increased system order. During and after the abrupt-change interval, NN-VSS-NLMS maintains a relatively lower MSE level compared with the other algorithms. After the transition, its steady-state error remains lower, while the fluctuation range is comparable to that of NP-VSS-NLMS and VSS-NLMS.

3.7. Performance Analysis Under Different SNR Conditions

To evaluate the robustness of the algorithms under different noise conditions, the proposed algorithm is compared with NLMS, NP-VSS-NLMS, and VSS-NLMS over an SNR range from 0 to 50 dB, as illustrated in Figure 16 and Figure 17 for filter orders of 8 and 32, respectively. When the SNR is 0 dB, the signal power is comparable to the noise power, and the step-size prediction of the NN becomes more sensitive to noise disturbances. As a result, the MSE curves exhibit relatively larger fluctuations, and the proposed algorithm performs slightly worse than NP-VSS-NLMS. As the SNR increases to 10–50 dB, the influence of noise is gradually reduced, leading to more stable step-size prediction. For $L=8$, the proposed algorithm exhibits a higher convergence rate and achieves a lower steady-state error compared with the other algorithms across most SNR conditions. For $L=32$, a similar tendency can be observed. As the SNR increases, all algorithms show improved convergence behavior and reduced steady-state error. NN-VSS-NLMS maintains a relatively faster convergence rate and a lower steady-state error, while the differences among the variable step-size algorithms become less pronounced compared with the case of $L=8$.

Although its performance is affected at very low SNR, the proposed algorithm exhibits clear advantages in medium and high SNR scenarios, which supports its applicability in practical acoustic signal processing tasks.

4. Conclusions

The proposed NN-VSS-NLMS algorithm improves adaptive filtering performance through learning-assisted step-size adjustment and demonstrates enhanced robustness under different signal conditions. By combining an analytically motivated reference step size with NN-based auxiliary adjustment, the algorithm achieves a fast convergence rate, reduced steady-state error, and strong tracking ability under stationary inputs, non-stationary inputs, and time-varying systems, thereby extending the applicability of traditional VSS-NLMS schemes to more challenging scenarios. The algorithm can be applied to practical signal-processing scenarios such as acoustic echo cancelation, active noise control, and system identification in environments with time-varying characteristics. Despite these advantages, several limitations should be acknowledged. First, the incorporation of an NN for step-size prediction introduces additional computational complexity, which may limit its applicability in large-scale or real-time scenarios. Second, the effectiveness of the NN-based prediction depends on the quality of the extracted features, and its performance may degrade under very low SNR conditions or severely corrupted sensor data. Third, although the algorithm exhibits good tracking ability in moderately time-varying systems, its adaptability to highly dynamic environments with abrupt changes may still be limited. Future work will focus on improving robustness under low SNR conditions by refining feature extraction and step-size control while reducing the network complexity to improve computational efficiency for real-time operation. Further evaluation in more realistic acoustic scenarios, including speech signals affected by nonlinear distortions or background interference, is expected to provide a more comprehensive understanding of the algorithm’s performance.