The Module Gradient Descent Algorithm via L₂ Regularization for Wavelet Neural Networks

Abstract

Although wavelet neural networks (WNNs) combine the expressive capability of neural models with multiscale localization, there are currently few theoretical guarantees for their training. We investigate the weight decay (${\mathrm{L}}_2$ regularization) optimization dynamics of gradient descent (GD) for WNNs. Using explicit rates controlled by the spectrum of the regularized Gram matrix, we first demonstrate global linear convergence to the unique ridge solution for the feature regime when wavelet atoms are fixed and only the linear head is trained. Second, for fully trainable WNNs, we demonstrate linear rates in regions satisfying a Polyak–Łojasiewicz (PL) inequality and establish convergence of GD to stationary locations under standard smoothness and boundedness of wavelet parameters; weight decay enlarges these regions by suppressing flat directions. Third, we characterize the implicit bias in the over-parameterized neural tangent kernel (NTK) regime: GD converges to the minimum reproducing kernel Hilbert space (RKHS) norm interpolant associated with the WNN kernel with ${\mathrm{L}}_2$. In addition to an assessment process on synthetic regression, denoising, and ablations across λ and stepsize, we supplement the theory with useful recommendations on initialization, stepsize schedules, and regularization scales. Together, our findings give a principled prescription for dependable training that has broad applicability to signal processing applications and shed light on when and why ${\mathrm{L}}_2$-regularized GD is stable and quick for WNNs.

1. Introduction

Wavelet neural networks (WNNs) are an appealing family of models because they combine multiscale, well-localized representations with the universal approximation capability of neural structures. Recent work ranges from graph-based wavelet neural networks to “decompose-then-learn” pipelines to deep models enhanced with wavelet transforms, demonstrating that wavelet structure can improve stability, efficiency, and interpretability across applications in vision, signal processing, and energy systems [1,2,3,4]. The optimization theory governing the training dynamics of WNNs is significantly less studied than that of other architectures, especially when gradient descent (GD) with L₂ regularization (weight decay) is employed. This gap between theoretical knowledge and real progress is the main topic of this research. Wavelet advantages in practice manifest along two axes: (a) multiscale localization, which captures local phenomena at various frequencies without losing broader context, and (b) computational efficiency when employing learnable or fixed wavelet filters, which reduces sensitivity to small perturbations in the data and speeds up training.

Baharlouei et al. categorize retinal optical coherence tomography anomalies using wavelet scattering, which has a lower processing cost and more robustness than more complex deep alternatives. Graph wavelet neural networks efficiently capture spatiotemporal dependencies, while other recent studies integrate wavelet decompositions into deep models for complex timeseries, such as wind power [5,6]. These empirical data pose a fundamental question: when and why does GD with L₂ converge steadily and rapidly for WNNs?

Two regimes make the interaction of L₂ with GD particularly noticeable: (1) the fixed-feature regime, in which only a linear head is trained and the wavelet dictionary is frozen; this results in ridge regression as the objective, which is substantially convex and hence admits global linear convergence rates. (2) The fully trainable regime, in which wavelet parameters (translations/dilations) and weights are tuned; in this case, generic tools to demonstrate linear rates within appropriate landscape regions are provided by conditions such as the Polyak–Łojasiewicz (PL) inequality [7,8]. L₂ regularization is one of the most used tools from an optimization perspective. L₂ has an implicit bias in deep networks in addition to its traditional use in conditioning and in imparting effective curvature (for example, for a linear head on fixed features). Weight decay encourages low-rank tendencies and chooses smaller-norm solutions in parameter or function space, as demonstrated by recent works. These phenomena are closely related to convergence speed and stability [9].

Another theoretical viewpoint is provided by the neural tangent kernel (NTK). Between 2022 and 2024, NTK theory became a solid foundation for understanding over-parameterized training dynamics. In this regime, a nonlinear network behaves linearly in function space around initialization with respect to an effective kernel, and GD with L₂ becomes equivalent to GD in the corresponding RKHS. Surveys and practical evaluations suggest that this perspective accounts for both the minimum-norm bias and convergence (via the kernel spectrum): L₂-regularized GD selects the minimum-RKHS-norm solution among all interpolants during interpolation [10]. Making this image unique to WNNs is our contribution, and we therefore ask: What is the appearance of the WNN-specific NTK with realistic initiation and limited dilation/shift ranges? What effects do wavelet selections have on the spectral constant governing the rate?

In-depth analyses of wavelet–deep integration reveal that training dynamics guarantees, especially with L₂, remain absent despite steady progress, and most contributions concentrate on architectural design or empirical advantages. Even with forward-looking hybrids like Wav-KAN (2024) [11], sharp statements concerning rates, step-size/regularization requirements, and stability bounds are still uncommon when compared to linear models or neural networks under restrictive assumptions. This disparity has practical implications: without principled direction, practitioners are forced to rely on costly trial-and-error to calculate the learning rate ($\eta$) and regularization ($\lambda$), and it becomes more challenging to ascertain why training is effective or ineffective.

This paper’s contributions are as follows. We provide a unified analysis of convergence for gradient descent on WNNs in three regimes with L₂ regularization:

(a)

A linear-head regime over a fixed wavelet dictionary. With explicit rates controlled by the eigenvalues of $( {\Phi }^T\Phi /n+\lambda I)$, we demonstrate global linear convergence to the unique ridge minimizer. This results in useful guidelines for considering ($\lambda$) as a conditioning lever instead of just an anti-overfitting knob.

(b)

WNNs that are fully trainable (nonconvex). GD converges to stationary points and enjoys linear rates within regions meeting PL under the conditions of natural smoothness and boundedness for wavelets (within a restricted dilation/shift domain). We give implementable step-size limitations and demonstrate how L₂ dampens flat directions to widen PL basins.

(c)

A regime that is over-parameterized (NTK). We extract rate constants related to the kernel spectrum induced by the wavelet dictionary and demonstrate that L₂ directs GD toward the minimum-RKHS-norm interpolant associated with the WNN-specific NTK.

To detail our practice and methodology, the following is a training regimen derived from our theory. To ensure stability without sacrificing fit, select dilations log-uniformly over a bounded range by sampling translations from the empirical input distribution; estimate a Lipschitz proxy to set ($\eta$); sweep ($\lambda$) over practical ranges; and monitor the near-constant contraction in gradient norms on a semi-log scale to track the onset of a linearconvergence phase. We also propose an evaluation protocol (synthetic approximation and denoising benchmarks) to support our theoretical claims and provide insight into regime transitions as ($\lambda$) and ($\eta$) change, in compliance with existing guidelines on convergence diagnostics and implicit regularization [12,13,14]. In doing so, we bridge a persistent gap between the empirical richness of wavelet-based models and the theoretical understanding of their training dynamics under L₂-regularized gradient descent.

We organize the remaining portions of this brief as follows In Section 2, we give a literature review. In Section 3, we provide an explanation of the pi-sigma network. In Section 4, we briefly describe a neural network model with the batch gradient method and smoothing regularization. Section 5 presents the main convergence theorem. Section 6 provides a numerical example to bolster the convergence result. We summarize the findings and conclusions in Section 7. We have relegated the proof of the theorem to the Appendix A.

2. Related Work

Wavelet neural networks (WNNs) are a hybrid of learnable parametric models and multiresolution signal analysis. Wavelet atoms (translations/dilations) are described as localized, multiscale features inside neural predictors and system identifiers in a 2025 topical review that focuses on WNNs for signal parameter estimation and next-generation wireless technology [15]. Wavelet scattering, which is a cascade of wavelet modulus/averaging with an analytical foundation, has demonstrated strong performance in biomedical imaging. Baharlouei et al. demonstrated better optical coherence tomography abnormality classification using wavelet scattering as opposed to heavier deep baselines, highlighting stability to noise and minor deformations [16]. Recent graph wavelet neural networks go beyond Euclidean grids by combining graph wavelet operators with temporal modeling to effectively capture localized spatio-temporal relationships; Wang et al. (2024) demonstrate improvements in accuracy and efficiency when mining rapidly changing social media graphs [17]. Wavelets’ joint time–frequency localization has been credited with the success of employing them as bases or activations inside PINNs for stiff nonlinear differential equations (such as Blasius flow) in physics-informed settings [18]. When taken as a whole, these lines of study encourage examining the relationship between optimization dynamics and wavelet structure during training. Recent hybrids have incorporated discrete wavelet transformations as differentiable layers to preserve high-frequency detail during down/up-sampling in deep nets, reducing aliasing and enabling reconstruction. This trend is also seen in segmentation and restoration models [19]. To improve interpretability and training speed, Wav-KAN (2024) integrates wavelet bases into KolmogorovArnold networks, unlike its MLP/Spl-KAN counterparts. Wav-KAN (2024) open-source implementations accelerate repeatability [11]. These studies show potential in reality, while often evaluating performance experimentally; they hardly ever define optimization landscapes or provide rates for first-order approaches under regularization.

Gradient technique convergence under PL/Kρ conditions. The Polyak–Łojasiewicz (PL) inequality ensures linear (geometric) convergence of gradient descent with suitable step sizes for nonconvex objectives with benign curvature, even in the absence of convexity. As a useful instrument for assessing contemporary learning issues and creating diagnostics (such gradient-norm contraction) in deep training, PL is consolidated in recent pedagogical notes and lecture compendia (2021, and 2023) [20,21]. Although PL has been used in a variety of network classes, there is still a lack of research on explicit PL verification for wavelet-parameterized objectives. Additionally, it is not yet clear how L₂ regularization alters PL basins in WNNs. Our PL-centric analysis with wavelet-boundedness assumptions and confined dilation/shift domains is motivated by this gap. Regarding implicit bias and weight decay (L₂), weight decay influences the implicit bias of gradient-based training in addition to explicit conditioning in linearized tasks (ridge). L₂ significantly changes optimization geometry and convergence behavior, as evidenced by recent theory (2024) that weight decay can increase generalization bounds and induce low-rank structure in learnt matrices [22]. Regularization is linked to faster and more stable descent along well-aligned routes, as demonstrated by related work that shows similar low-rank effects in attention layers where multiplicative parameterizations interact with L₂ to prefer compact spectra [23]. However, these findings have not been thoroughly examined in relation to WNNs, where wavelet dilations and translations are among the parameters. They are established for ReLU/transformer-style architectures.

In an RKHS controlled by the neural tangent kernel (NTK), network training follows kernel gradient descent at initialization and at large width. This results in minimum-norm interpolation and spectral-rate predictions under explicit/implicit regularization. When NTK accurately forecasts optimization trajectories and generalization is described in surveys and empirical analyses extensions adjust NTK to surrogate gradient regimes and non-smooth activations [24,25]. A WNN-specific NTK that takes parameterized wavelet atoms into account has not yet been fully developed, despite the fact that NTK has been calculated for popular MLP/CNN designs. Our work explains how L₂ drives convergence to the minimum-RKHS-norm interpolant caused by the wavelet dictionary and initialization and offers a specialization of the NTK perspective on WNNs.

Convergence restrictions have been added to unrolled optimization algorithms in recent years to increase stability. Using learnable step sizes and a monotonic descent constraint, Zheng et al. [26] presented a deep unrolling method using a proximal gradient descent framework. The convergence of gradient descent techniques with regularization in wavelet neural networks is relevant, and their theoretical study demonstrates that such limitations can ensure convergence behavior. Additionally, iterative regularization techniques have been used to examine the theoretical foundations of regularization effects in neural network training. In the context of inverse problems, Cui et al. [27] showed that unfolding iterative algorithms can be used as regularization strategies, guaranteeing stability and convergence. This viewpoint supports the notion that incorporating L₂ regularization into gradient descent enhances convergence to stable solutions by acting as an iterative regularization procedure.

Despite the fact that most research is focused on specific applications or general neural network topologies, the field of gradient descent with L₂ regularization in wavelet neural networks allows for the synthesis of theoretical insights from various studies. With carefully designed gradient descent algorithms improved by L₂ regularization, wavelet neural network training may achieve stable and efficient convergence, according to the research’ explanations of acceleration strategies, implicit regularization effects, and convergence limitations. This convergence is further improved by the theoretical frameworks created for iterative regularization and unrolled optimization approaches, which also provide a foundation for further research into specialized structures like wavelet neural networks.

3. Preliminaries and Problem Setup

3.1. Wavelet Neural Network (WNN) Model

Let $\psi : {\mathbb{R}}^d\to \mathbb{R}$ be a mother wavelet. For parameters ${\theta }_j=(a_j,b_j)$ with dilation ${\mathrm{a}}_{\mathrm{j}}>0$ and translation $b_j\in {\mathbb{R}}^d$, define the atom ${\phi }_{\mathrm{j}}({\mathcal{x}}_{\mathrm{i}};{\theta }_{\mathrm{j}})=\psi \left(\right(\mathcal{x}-{\mathrm{b}}_{\mathrm{j}})/{\mathrm{a}}_{\mathrm{j}})$. A single-hidden-layer WNN with m atoms outputs $\mathcal{z}(\mathcal{x};\mathcal{W},\Theta )=\sum _{j=1}^m{\mathcal{w}}_j {\phi }_j({\mathcal{x}}_i;{\theta }_j)$. Given data ${\left\{\right(x_i,y_i\left)\right\}}_{i=1}^n, \mathrm{let} \Phi \in {\mathbb{R}}^{n\times m}$ with ${\Phi }_{ij}={\phi }_j({\mathcal{x}}_i;{\theta }_j)$.

$${\varphi }_j\left(\mathcal{x};{\theta }_j\right)=\psi \left({\displaystyle \frac{x-b_j}{a_j}}\right), {\theta }_j=\left(a_j,b_j\right)$$

$$\mathcal{z}\left(\mathcal{x};\mathcal{W},\Theta \right)=\sum _{j=1}^m{\mathcal{w}}_j{\theta }_j\left(x;{\theta }_j\right)$$

3.2. Assumptions (Wavelets, Data, and Loss)

Wavelet atoms at multiple dilations refer to a mathematical concept that extends the standard wavelet transform by using a more complex set of scaling and transforming functions to analyze signals. For example, for the Mexican hat wavelet or Marr wavelet (see Figure 1), wavelet atoms are obtained by scaling the Mexicanhat mother wavelet:

$$\psi \left(\mathcal{x}\right)=\left(1-x^2\right)e^{-x^{1/2}}$$

The curves show ${\psi }_a\left(\mathcal{x}\right)=a^{-{\displaystyle \frac{1}{2}}}\psi \left({\displaystyle \frac{\mathcal{x}}{a}}\right), x\in \left\{-5,+5\right\}$ for dilations $a\in \left\{0.5, 1, 2, 4\right\}$. The number of points (N = 2000) allows for the drawing of smooth, high-precision curves. The amplitude normalization $a^{-1/2}$ preserves the L₂ energy across scales.

Figure 1. Wavelet atoms at multiple dilations (Mexican hat).

Here, we adopt standard assumptions:

• (A1) $\psi$ is twice continuously differentiable with bounded value, gradient, and Hessian;
• (A2) dilations/translations are bounded ($0<b_{min}\le b_j\le b_{max} \mathrm{and} ‖a_j‖\le A$);
• (A3) inputs lie in a compact set;
• (A4) the loss is smooth (and strongly convex in its first argument for squared loss);
• (A5) feature Jacobians w.r.t. $\theta$ are uniformly bounded. Under (A1–A5), $\nabla \mathcal{L}$ is Lipschitz on the feasible set; in particular, the head-only objective in $\mathcal{W}$ is strongly convex due to $\lambda$.

3.3. Training Algorithms (GD/SGD with Weight Decay)

We minimize the L₂-regularized empirical risk:

$$\mathcal{L}\left(\mathcal{W},\Theta \right)={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\left(\mathcal{z}\left(x_i;\mathcal{W},\Theta \right)-y_i\right)}^2+{\displaystyle \frac{\lambda }{2}}\left({\left|\mathcal{W}\right|}_2^2-{\left|\Theta \right|}_2^2\right)$$

With squared loss, the gradient w.r.t. $\mathcal{W}$ is

$${\nabla }_W\mathcal{L}={\displaystyle \frac{1}{n}}\left(\theta \mathcal{W}-y\right)+\lambda \mathcal{W}$$

3.4. Problem Decompositions: Three Regimes

• (R1) Fixed-feature (linear head/ridge). Freezing $\Theta$ reduces the problem to ridge regression, with Hessian $\mathcal{H}=({\Phi }^{\mathrm{T}}\Phi /n + \lambda I) ⪰ \lambda I$, and GD enjoys global linear convergence for $0<\eta <2/{\lambda }_{max\left(\mathcal{H}\right)}$.

$$\underset{\mathcal{W}}{\min }{\displaystyle \frac{1}{2n}}{\left|\theta \mathcal{W}-y\right|}_2^2+{\displaystyle \frac{\lambda }{2}}{\left|\mathcal{W}\right|}_2^2$$

• (R2) Fully trainable WNN (nonconvex). Both $\mathcal{W}$ and Θ are updated; we obtain convergence to stationary points and linear phases under a Polyak–Łojasiewicz (PL) inequality on $\mathcal{L}$.

$${\displaystyle \frac{1}{2}}{\left|\nabla \mathcal{L}\left(\theta \right)\right|}_2^2\ge {\mu }_{PL}\left(\mathcal{L}\left(\theta \right)-{\mathcal{L}}^{\ast }\right)$$

• (R3) Over-parameterized (NTK/linearization). For large m and small $\eta$, dynamics linearize around initialization; function updates follow kernel GD with a WNN-specific NTK K:

$${\mathcal{z}}_{t+!}={\mathcal{z}}_t-\eta k({\mathcal{z}}_t-y)$$

3.5. PL Inequality and Its Role

If L satisfies a PL inequality with constant ${\mu }_{PL}$ on a domain $\mathcal{D}$ and $\nabla L$ is $\mathcal{L}$-Lipschitz, then for $0<\eta \le 1/\mathcal{L}$, gradient descent yields geometric decay $\mathcal{L}\left({\theta }^t\right)-{\mathcal{L}}^{\ast } \le {(1-\eta {\mu }_{PL})}^t\left(\mathcal{L}\right({\theta }^0)-{\mathcal{L}}^{\ast })$. In WNNs, L₂ enlarges PL regions by damping flat directions along scale/shift parameters.

3.6. NTK for WNNs (Overview)

The neural tangent kernel (NTK) is a kernel function that describes how a neural network’s output changes during training with gradient descent. It allows for the theoretical analysis of neural network training by reframing the problem as a kernel method, especially in the limit of infinite width where the kernel becomes constant during training. The NTK is calculated as the dot product of the gradients of the network’s output with respect to its parameters for two different inputs. Let K denote the NTK at initialization. With small learning rates and large width, training follows kernel GD in the RKHS induced by K; convergence speed is governed by the spectrum $\left\{{\lambda }_i\right(K\left)\right\}$. With L₂, GD converges to the minimum-RKHS-norm solution among all interpolants.

Let the WNN output be parameterized by $\theta$ (all trainable parameters). For an input $x,$ the network output is $f(x;\theta )$. At initialization ${\theta }_0,$ the NTK matrix $K\in R^{N\times N}$ on inputs ${\left\{x_i\right\}}_{i=1}^N$ is

$$K_{ij}={{\nabla }_{0}f(x_i;{\theta }_0)}^T{\nabla }_{0}f(x_j;{\theta }_0)$$

Equivalently, if $J\in R^{N\times P}$ is the Jacobian whose i-th row is $J_i={{\nabla }_{0}f(x_i;{\theta }_0)}^T$, then $K=J{J}^T$.

Using Figure 2, we can visualize how the NTK evolves during gradient descent. The input grid has N = 150 points uniformly spaced on [0, 1]. The WNN architecture has a single hidden layer with M = 30 wavelet atoms. The parameter vector $\theta$ has length P = 3. Initialization (used to compute NTK at init) features amplitudes $c_j~\mathcal{N}\left(0,{0.5}^2\right)$, scales $a_j~\mathcal{N}\left(1.0,{0.1}^2\right)$, and translations $b_j~U\left(\mathrm{0,1}\right)$. The full $N\times N$ NTK matrix is used to produce Figure 2. This figure plots the eigenvalues of the NTK for the WNN, typically on a log scale, showing how the spectrum decays across modes.

Figure 2. WNN-NTK eigenvalue decay (illustrative).

Figure 2 reveals different critical advancements of studying NTK eigenvalue behavior in wavelet neural networks, especially under L₂ regularization. Figure 2 is essential because it visually and mathematically explains why WNNs train faster, converge more consistently, and benefit more from L₂ regularization than conventional networks. It links optimization theory, NTK spectral properties, convergence behavior and multiscale feature learning.

$$K_{toy}=\left[\begin{array}{cccccc}1.20& 0.83& 0.45& 0.12& 0.05& 0.02\\ 0.83& 1.30& 0.76& 0.22& 0.07& 0.03\\ 0.45& 0.76& 1.10& 0.48& 0.18& 0.06\\ 0.12& 0.22& 0.48& 0.98& 0.54& 0.02\\ 0.05& 0.07& 0.18& 0.54& 0.95& 0.48\\ 0.02& 0.03& 0.06& 0.20& 0.48& 0.88\end{array}\right]$$

3.7. Step-Size and Regularization Prescriptions (Preview)

• Head-only (R1): choose $\eta \in (0, 2/\mathcal{L})$ with $\mathcal{L}={\lambda }_{max\left(\mathcal{H}\right)}$; increasing λ raises $\mu ={\lambda }_{max\left(\mathcal{H}\right)}$ and improves the condition number $\mathcal{L}/\mu$.
• Full WNN (R2): use conservative $\eta \le 1/{\mathcal{L}}_{emp}$ (empirical Lipschitz proxy); increase $\lambda$ if gradient-norm contraction stalls.
• NTK (R3): stable $\eta <2/{\lambda }_{max\left(K\right)}$; L₂ controls norm growth and selects the minimum-RKHS-norm interpolant.

Gradient-norm decay (linear phase evident on semi-log Figure 3). This corresponds to a typical semi-log plot showing that decays approximately exponentially in $t$.

$$‖\nabla \mathcal{L}\left({\theta }_t\right)‖$$

Figure 3. Gradient-norm decay (linear phase evident on semi-log).

Target function:

$$f\left(x\right)=\sin \left(2\pi x\right)$$

where there are N = 100 samples uniformly spaced on $[0, 1]$. A random Gaussian initialization ${\mathcal{W}}_0\left[i\right]~\mathcal{N}\left(0,{0.05}^2\right)$ has a learning rate $\eta =0.1$, regularization $\lambda =\left\{0,1\times {10}^{-4},1\times {10}^{-3}\right\},$ and 200 epochs.

The curves in Figure 3 show different significant improvements in L₂ regularization. Linear decline indicates exponential convergence on a semi-log plot. Gradient magnitudes decrease more quickly, the slope of log $(\Vert \nabla L\Vert )$ becomes steeper, and the optimization phase enters the linear convergence area earlier when L₂ regularization is used. In the unregularized situation ($\lambda = 0$), plateaus are frequent, gradient norms frequently oscillate, and weak curvature causes noise amplification. When ($\lambda >0$), curves exhibit more steady descent, no abrupt oscillations, and smooth monotonic decay. This illustrates how L₂ regularization reduces saddle-point sensitivity, eliminates flat areas, and enforces strong convexity in the local basin to stabilize the loss landscape.

4. Mmethodology

4.1. Objective and Gradient Updates

We consider the L₂-regularized objective and its gradients w.r.t. $\mathcal{W}$ and $\Theta$.

$$\mathcal{L}\left(\mathcal{W},\Theta \right)={\displaystyle \frac{1}{n}}\sum _{i=1}^n𝓁\left[\mathcal{z}\left({\mathcal{x}}_i;\mathcal{W},\Theta \right),y_i\right]+{\displaystyle \frac{\lambda }{2}}\left({\left|\mathcal{W}\right|}_2^2+{\left|\Theta \right|}_2^2\right)$$

The first-order updates (vanilla GD) read

$${\mathcal{W}}^{t+1}={\mathcal{W}}^t-\eta {\nabla }_{\mathcal{W}}\mathcal{L}\left({\mathcal{W}}^t,{\Theta }^t\right)$$

$${\Theta }^{t+1}={\Theta }^t-\eta {\nabla }_{\Theta }\mathcal{L}\left({\mathcal{W}}^t,{\Theta }^t\right)$$

4.2. Fixed-Feature (Ridge) Training of the Linear Head

Freezing Θ reduces training to ridge regression on Φ; the regularized Hessian $\mathcal{H}$ is wellconditioned for $\lambda >0$.

$$\underset{W}{\mathit{min}}{\displaystyle \frac{1}{2n}}\left|\Theta \mathcal{W}-y\right|+{\displaystyle \frac{\lambda }{2}}{\left|\mathcal{W}\right|}_2^2$$

$${\left|{\mathcal{W}}^{t+1}-{\mathcal{W}}^t\right|}_2\le {\rho }^t{\left|{\mathcal{W}}^0-{\mathcal{W}}^{\ast }\right|}_2, \rho =\underset{i}{\mathit{max}}\left|1-\eta {\lambda }_i\left(\mathcal{H}\right)\right|$$

$$K\left(\mathcal{H}\right)={\displaystyle \frac{{\lambda }_{max}\left(\mathcal{H}\right)}{{\lambda }_{min}\left(\mathcal{H}\right)}}={\displaystyle \frac{\left({\sigma }_{max}^2/n\right)+\lambda }{\left({\sigma }_{min}^2/n\right)+\lambda }}$$

4.3. Fully Trainable WNN: Block GD, Schedules, and Stability

Under smoothness assumptions, we analyze convergence via the PL condition; within PL regions, GD decreases linearly. We adopt practical schedules for $\eta$ (constant, step, cosine) and a Polyak-type step when a reliable target is available. We use the target function the same as in Section 3.7, with 100 points. The training parameters are as follows: constant $\eta =0.01$, decay at steps 50 and 100, and cosinegiven by

$${\lambda }_t={\displaystyle \frac{{\lambda }_0}{2}}\left(1+cos\left({\displaystyle \frac{\pi t}{T}}\right)\right)$$

Study stability and convergence under different LR schedules are plotted in Figure 4.

Figure 4. Learning rate schedules (constant, step, cosine).

4.4. Choosing η and λ: Prescriptions and Diagnostics

• R1: choose $\eta \in (\mathrm{0,2}/\lambda \_max(\mathcal{H}\left)\right)$ and sweep $\lambda$ logarithmically.
• R2: use $\eta \le 1/\mathcal{L}\_emp$ and increase λ if gradient-norm contraction stalls.
• R3: ensure $0<\eta <2/{\lambda }_{max\left(K\right)}$; L₂ controls norm growth and selects the minimum-RKHS-norm interpolant.

5. Theoretical Results

5.1. Linear Convergence for the Fixed-Feature (Ridge) Regime

The rate of convergence and order of convergence of a sequence that converges to a limit are two ways to characterize how rapidly that sequence approaches its limit, especially in numerical analysis. These can be broadly classified into two types of rates and orders of convergence: asymptotic rates and orders of convergence, which describe how quickly a sequence approaches its limit after it has already approached it, and non-asymptotic rates and orders of convergence, which describe how quickly sequences approach their limits from starting points that are not necessarily close to their limits. Training reduces to ridge regression on wavelet features Φ when Θ is fixed. Below, let S stand for the regularized Hessian.

$$\mathcal{H}={\displaystyle \frac{1}{n}}{\Phi }^T\Phi +\lambda I$$

For step sizes $0<\eta <2/{\lambda }_{max\left(\mathcal{S}\right)}$, gradient descent converges linearly to the unique minimizer ${\mathcal{W}}^{\ast }$. The error contracts at a rate controlled by the spectrum of $\mathcal{H}$ in Figure 5. The function used is a linear model (ridge regression $f_3\left(x\right)$):

$$y=X{\mathcal{W}}^{\ast }$$

where

$${\mathcal{W}}^{\ast }=\left[\begin{array}{c}2\\ -0.5\\ 0.5\end{array}\right]$$

Figure 5. Linear convergence in the ridge: theoretical ${\rho }^t$ vs. observed residual decay (semilog).

The error contracts at a rate controlled by the spectrum of $\mathcal{H}$:

$${\left|{\mathcal{W}}^{t+1}-{\mathcal{W}}^t\right|}_2\le {\rho }^t{\left|{\mathcal{W}}^0-{\mathcal{W}}^{\ast }\right|}_2, \rho =\underset{i}{\mathit{max}}\left|1-\eta {\lambda }_i\left(\mathcal{H}\right)\right|$$

Figure 5 evaluates linear convergence using a simple linear model $\left(f\right(x)=2x+1)$ with $\eta =0.01,$ $\lambda =0.1$, and $\rho =0.998$.

5.2. Fully Trainable WNN Under a PL Inequality

We assume smoothness and boundedness of wavelet atoms over a constrained dilation/shift domain. If $\mathcal{L}$ satisfies a PL inequality with constant ${\mu }_{PL}$ and $\nabla \mathcal{L}$ is $\mathcal{L}$-Lipschitz, then GD with $0<\eta \le 1/\mathcal{L}$ enjoys a linear decrease in the objective:

$${\displaystyle \frac{1}{2}}{\left|\nabla \mathcal{L}\left(\theta \right)\right|}_2^2\ge {\mu }_{PL}\left|\mathcal{L}\left(\theta \right)-{\mathcal{L}}^{\ast }\right|⟹\mathcal{L}\left({\theta }^t\right)-{\mathcal{L}}^{\ast }\le {\left(1-\eta {\mu }_{PL}\right)}^t\left(\mathcal{L}\left({\theta }^0\right)-{\mathcal{L}}^{\ast }\right)$$

PL-like regions are enlarged, and stability is enhanced by L₂ regularization, which intuitively dampens flat directions linked to scale/shift redundancy. The impact of λ on landscape smoothness is shown in Figure 6, where larger λ increases the size of benign (PL-like) areas. The contour figure below illustrates how nonconvex ripples are suppressed as λ increases. This type of figure visualizes how the loss landscape changes as the ridge (L₂) regularization parameter λ increases, using a small WNN and a simple 1-D regression function:

$$f\left(x\right)=\sin \left(2\pi x\right)+0.3\mathrm{c}\mathrm{o}\mathrm{s}\left(4\pi x\right)$$

where

$$x_i~Uniform\left(\mathrm{0,1}\right), i=1,\dots ,100$$

$$y_i=f\left(x_i\right)$$

Figure 6. Effect of λ on Landscape Smoothness.

A small wavelet network is used to make landscape plots feasible: there is one input dimension and six wavelet neurons, and the mother wavelet is Mexican hat. To show how L₂ regularization smooths the loss landscape, we vary λ values: $\lambda \in \left\{0, 0.01, 0.1, 1\right\}$.

Figure 6 shows how the shape of the optimization landscape for WNNs is affected by changing the L₂ regularization parameter λ. The loss surface is shown in the picture, along with areas where the function acts in accordance with a PL inequality—a requirement that is known to ensure global linear convergence of gradient descent.

A 2-D slice or contour of the loss over a tiny parameter plane (two directions in parameter space) is shown in each panel of Figure 7. The L₂ regularization λ is the only difference between panels. Typical observations (compare small → large λ). For λ = 0 (no regularization) the surface is rough, with several tiny ridges/valleys and stronger curvature in areas. Contours are uneven and anisotropic. There are saddle-like landforms and little basins. When λ (e.g., 1 × 10⁻² → 0.1), the loss surface gets notably rounder and more convex-looking in this slice. Narrow wells are filled in and the central basin grows. The spacing between contours is more consistent (gradients vary gently). While for big λ (e.g., 1.0), the regularization term dominates; the surface is exceedingly smooth and globally similar to a quadratic (the λ‖w‖² term). The basin is large and steep toward center; little nonconvex ripples are mostly gone.

Figure 7. 2D Loss Landscape with three different values of λ.

6. Experiments and Evaluation

6.1. Datasets and Tasks

We evaluate wavelet neural networks (WNNs) with L₂-regularized gradient descent across three complementary experimental settings, each designed to isolate a different aspect of optimization, generalization, and landscape behavior.

(A) One-Dimensional Synthetic Function Approximation (Optimization Analysis)

To study optimization dynamics under full control of ground truth and curvature, we employ three canonical synthetic regression tasks, each selected to emphasize a different structural property relevant to wavelets and convergence theory.

(1)

Smooth Low-Frequency Signal

$$f_1\left(x\right)=\sin \left(2\pi x\right)+0.3\mathrm{c}\mathrm{o}\mathrm{s}\left(4\pi x\right)$$

Samples: N = 500 uniformly spaced points

(2)

Localized Bump Function

$$f_2\left(x\right)=exp\left(-{\displaystyle \frac{{\left(x-0.3\right)}^3}{0.02}}\right), x\in [-1,1]$$

Samples: N = 400.

(3)

Ridge/Non-Smooth Function

$$f_3\left(x\right)=\left|\mathrm{x}\right|, x\in [-1,1]$$

Samples: N = 300.

(B) Image-Like Denoising Task (PSNR vs. Noise Level σ)

To evaluate practical generalization and robustness, we construct image-like signals using 2D tensorized wavelets.

The dataset consists of the following:

Synthetic Image Function

$$f_4\left(x_1,x_2\right)=\sin \left(\pi x_1\right)+0.3\cos \left(\pi x_1\right), \left(x_1,x_2\right)\in {[-1,1]}^2$$

where the grid has 50 × 50 points and the noise model is

$$y=f_4+ϵ, ϵ~\mathcal{N}\left(0,{\sigma }^2\right)$$

Noise levels are $\sigma =0.01, 0.05, 0.1, 0.2$, and PSNR (peak signal-to-noise ratio) is the chosen metric.

6.2. Metrics

We report mean squared error (MSE) and peak signal-to-noise ratio (PSNR).

$$MSN={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\left(y_i-{\widehat{y}}_i\right)}^2$$

$$PSNR=10{\log }_{10}\left({\displaystyle \frac{{MAX}_i^2}{MSE}}\right)$$

6.3. Synthetic Regression (Approximation)

The WNN captures the target function with small bias and controlled variance. Using localized bump function ($f_2\left(x\right)$), the overlay in Figure 8 compares ground truth vs. prediction. Figure 8 persuasively shows that L₂ regularization improves training stability and prevents overfitting; WNNs provide superior approximation of functions with localized and multiscale behavior; and the WNN learns a smooth, accurate, and well-conditioned outcome while preserving the structure that comprises the underlying function.

Figure 8. Synthetic regression: ground truth vs. WNN prediction.

6.4. Denoising Robustness

The capacity of a model to sustain consistent performance in the face of noise, whether from adversarial attacks or natural sources like picture noise, is known as denoising robustness. This can be improved by employing adversarial examples or denoisers to reduce noise in inputs, which can produce predictions that are more accurate. Figure 9 shows the PSNR after simulating additive noise with standard deviation $\sigma$ by using the smooth low-frequency signal ($f_1\left(x\right)$) function. Stronger structure preservation under noise is indicated by the WNN’s greater PSNR across σ levels.

Figure 9. PSNR vs. σ: WNN vs. baseline CNN.

6.5. Sensitivity to Learning Rate and Weight Decay

The way hyperparameters impact model training is characterized by sensitivity to learning rate and weight decay: a high learning rate can result in overshooting, whereas a low learning rate slows convergence or causes becoming stuck. Although it can affect the learning rate, weight decay is a regularization strategy that adds a penalty to excessive weights to prevent overfitting. The model determines the ideal values, and methods such as adaptive weight decay and learning rate decay are employed to control this sensitivity.

In Figure 10, we sweep $\eta \in [1\times {10}^{-4},1\times {10}^{-1}]$ and $\lambda \in [1\times {10}^{-6},1\times {10}^{-2}]$ logarithmically. A broad valley of lowvalidation MSE appears near $(\eta \approx 3\times {10}^{-3}, \lambda \approx 3\times {10}^{-4})$. Figure 10 shows a plot created using the smooth low-frequency signal ($f_1\left(x\right)$) function.

Figure 10. Sensitivity heatmap: validation MSE vs. ($\eta , \lambda$).

6.6. Learning Dynamics

The purpose of this subsection is to evaluate the denoising ability of WNN; to do so, we use the following function:

$$y=f_4+ϵ, ϵ~\mathcal{N}\left(0,{\sigma }^2\right)$$

Learning curves (train/val) show a clear linear phase on a semi-log scale, consistent with PL-based analysis (see Figure 11). No overfitting is observed over the epochs that are shown.

Figure 11. Learning curves (train/val).

6.7. Prediction Fidelity

The degree to which the model’s predictions match the actual data is known as fidelity. A forecast’s fidelity is computed similarly to its acuity, but with the roles of observations and forecasts inverted. To carry out a side-by-side comparison of noisy vs. ground truth vs. reconstructed, we use the following function:

$$y=f_4+ϵ, ϵ~\mathcal{N}\left(0,{\sigma }^2\right)$$

It is evident from Figure 12 that there is little bias and controlled variance because the distribution of predictions versus ground truth is clustered close to the identity line.

Figure 12. Scatter: predictions vs. ground truth.

6.8. Reproducibility Checklist

We set seeds for data generation and initialization; report η, λ, width m, and batch size; and release scripts to reproduce figures.

7. Discussion and Limitations

7.1. Practical Implications

Our analysis advocates viewing $\lambda$ as a conditioning lever rather than only a regularization knob: a small λ value risks ill-conditioning and slow convergence; overly large λ values bias solutions excessively and harm accuracy. A U-shaped validation error curve is expected as λ varies (see Figure 13). There are sinusoidal and small high-frequency spikes, and the parameters are constant LR vs. cosine LR. We track gradient norms over iterations:

Figure 13. Validation error vs. λ (bias–variance trade-off).

7.2. Sensitivity and Stability

Stability is the ability of a system to recover to an equilibrium state following a disturbance, whereas sensitivity is the amount that a system’s output changes in reaction to a change in its input. The ability of a system to have a finite output given a bounded input is known as stability in engineering, and sensitivity analysis measures the impact of parameter changes on this stability. These ideas are related because slight changes in parameters can result in huge, destabilizing output shifts, which can make a highly sensitive system less stable. Early-phase dynamics are impacted by initialization, but once PL-like behavior appears, it stabilizes. Weight decay dampens flat directions, reducing variance in trajectories; a slight spread among random seeds is normal. Figure 14 below illustrates how stability and sensitivity are avoided.

Figure 14. Initialization sensitivity: validation MSE across runs.

7.3. Robustness Under Distribution Shift

To show how WNN captures sharp features compared to MLP/CNN, we use a highly localized “peak”:

$$f\left(x\right)=exp(-100{(x-0.5)}^2)$$

The capacity of a model to continue performing as the statistical characteristics of the data it meets in the actual world differ from those of its training data is known as robustness under distribution. Wavelet localization preserves important patterns and increases robustness to moderate covariate fluctuations (Figure 15). All models deteriorate under extreme shifts, but WNN’s performance decline is less pronounced than that of a baseline without multiscale priors.

Figure 15. Robustness under shift (performance vs. severity).

7.4. Limitations

Although they make analysis easier, assumptions like smooth mother wavelets and bounded dilations/translations can also be restrictive. Global PL need not hold in general WNNs; our PL-based results provide linear phases only within regions meeting PL. Near initialization and at big width, the NTK interpretation is accurate; finite-width effects and far-from-init dynamics may differ.

7.5. Future Work

Future work will include deriving explicit WNN-specific NTKs for common wavelet families; tighter conditions under which L₂ induces PL globally; adaptive schedules that jointly tune η and λ (see Table 1); and extensions to classification losses and structured outputs.

Table 1. Ablations (illustrative).

η	λ	Val MSE	PSNR (dB)
3 × 10−3	3 × 10−4	0.032	31.2
1 × 10−3	1 × 10−4	0.036	30.8
5 × 10−3	1 × 10−3	0.038	30.1

8. Conclusions and Future Directions

This study used controlled hyperparameter sweeps, NTK-based interpretation, and synthetic benchmarks to provide a thorough examination of neural network training under L₂ regularization. We confirmed the L₂ penalty’s sparsity-inducing effect through function approximation tasks, and we consistently saw improvements in gradient stability and convergence behavior. The image-like denoising studies further showed that L₂ regularization achieves competitive PSNR while avoiding overfitting by effectively balancing reconstruction accuracy and model compactness. Together, our theoretical and empirical analyses of loss decay, gradient-norm evolution, weight-norm trajectories, and NTK eigenvalue analysis demonstrated that the L₂ penalty encourages structured sparsity without compromising the underlying model’s expressive power. However, the method requires careful calibration to ensure stability because it is sensitive to the choice of regularization weight λ and learning rate η. All things considered, the findings validate that L₂ regularization is a viable method for constructing sparse, effective neural networks with well-behaved optimization landscapes. These results may be extended to deeper architectures, adaptive regularization schedules, and higher-dimensional data in future studies. Additionally, the theoretical relationships between sparsity and NTK dynamics may be further investigated.

Future Directions

(a)

For canonical wavelet families (Mexican hat, Morlet, and Daubechies), we will derive closed-form WNN-specific NTKs and examine their spectra with realistic initializations.

(b)

In order to quantify expansion as a function of λ, we will determine the conditions under which L₂ causes global or broader PL regions for trainable dilations/translations.

(c)

We will look to create adaptive controllers with theoretical stability guarantees that simultaneously adjust η and λ utilizing real-time spectral/gradient diagnostics.

(d)

We will use wavelet priors to expand the analysis to structured outputs (such as graphs and sequences) and classification losses (logistic and cross-entropy).

(e)

We will examine robustness in the presence of adversarial perturbations and covariate shift, when wavelet localization might provide demonstrable stability benefits.