Hausdorff Difference-Based Adam Optimizer for Image Classification

Abstract

To address the limitations of fixed-order update mechanisms in convolutional neural network parameter training, an adaptive parameter training method based on the Hausdorff difference is proposed in this paper. By deriving a Hausdorff difference formula that is suitable for discrete training processes and embedding it into the adaptive moment estimation framework, a generalized Hausdorff difference-based Adam algorithm (HAdam) is constructed. This algorithm introduces an order parameter to achieve joint dynamic control of the momentum intensity and the effective learning rate. Through theoretical analysis and numerical simulations, the influence of the order parameter and its value range on algorithm stability, parameter evolution trajectories, and convergence speed is investigated, and two adaptive order adjustment strategies based on iteration cycles and gradient feedback are designed. The experimental results on the Fashion-MNIST and CIFAR-10 benchmark datasets show that, compared with the standard Adam algorithm, the HAdam algorithm exhibits clear advantages in both convergence efficiency and recognition accuracy.

1. Introduction

Convolutional neural networks (CNNs) have established a dominant position in tasks such as image classification [1], object detection [2], semantic segmentation [3], and scene understanding [4] owing to the structural advantages of local receptive fields, weight sharing, and hierarchical feature extraction. By utilizing end-to-end training mechanisms, CNNs are capable of learning multiscale representations automatically, ranging from low-level edge textures to high-level semantic concepts directly from two-dimensional image data [5]. Consequently, the reliance on traditional manual feature engineering is fundamentally alleviated. In recent years, driven by the substantial increase in the scale of training data and the enhancement of computational power, network architectures have evolved towards greater depth, width, and topological complexity. This evolution has continuously elevated the accuracy limits of large-scale image recognition. Nevertheless, the expansion of the network scale results in an optimization landscape of the loss function that exhibits complex characteristics, such as high-dimensional non-convexity, heterogeneous curvature, and multiple local extrema. These complexities impose significantly stringent challenges on the convergence efficiency, numerical stability, and generalization ability of training algorithms.

In the training process of deep convolutional networks [6], optimization algorithms govern the update trajectory and step size of parameters within the solution space. These algorithms serve as a critical factor determining the convergence speed and generalization performance of the model. Although Stochastic Gradient Descent (SGD) [7] and the variants with momentum constitute the cornerstone of deep training, such methods often rely on globally unified learning rates. Consequently, such methods are sensitive to hyperparameters and prone to stagnation in saddle points or flat regions on high-dimensional complex loss surfaces. To address these issues, adaptive optimization algorithms have been developed to implement dynamic adjustments of learning rates for individual parameters by introducing first-order and second-order gradient statistics. Among these, adaptive moment estimation (Adam) [8] is widely adopted as the default optimizer for CNN training due to the simplicity of the structure and rapid convergence. However, theoretical analysis and practical experience indicate that the standard Adam algorithm is intrinsically established on an update mechanism based on integer-order differences. When accurate gradient information is sparse or the loss landscape is complex, this fixed integer-order rule often fails to characterize the complex dynamics of deep networks adequately. This limitation frequently leads to oscillating update trajectories and restricted generalization performance. While several variants of Adam—AdamW, RAdam, and AdaBelief—have been proposed to address specific limitations—such as decoupled weight decay, variance rectification, or curvature-aware steps—these methods generally retain the classical integer-order update structure. In contrast, this paper seeks to fundamentally generalize the underlying difference operator. By incorporating the Hausdorff difference, we introduce a fractal time-scale mechanism that inherently modulates the trade-off between historical memory and current gradient responsiveness. This approach offers a generalized framework where standard Adam is merely a special case.

To exploit historical gradient information more fully and capture long-range dependencies and nonlocal effects during parameter evolution, the introduction of fractional calculus theory has emerged as a significant research direction in optimization algorithms [9]. Unlike integer-order models that rely solely on instantaneous rates of change [10], fractional-order models inherently possess properties of memory and nonlocality by introducing a continuously tunable order parameter. This inclusion allows for a more refined description of the dynamic behavior of complex systems. Embedding the order parameter into parameter update rules essentially constructs a tunable trade-off mechanism between the current gradient response and the historical cumulative influence [11]. Existing studies have confirmed that a rational configuration of the order helps to enhance the stability of the algorithm [12], smooth the parameter evolution trajectory, and improve the capability of the model to fit complex data distributions. Therefore, the introduction of fractional-order concepts into CNN optimization possesses a clear theoretical basis and application value [13].

Fractional derivatives find extensive applications in modeling, spanning diverse fields such as physics [14], engineering [15], image processing [16], and economics [17]. Although classical fractional derivatives offer modeling advantages [18], the definitions typically involve convolution operations over long historical intervals. This requirement results in computational complexity for a single iteration that increases significantly with time [19]. In the scenario of large-scale training for deep networks [20], this computational bottleneck severely restricts the practical application of fractional-order methods. To control computational costs while retaining memory characteristics, generalized calculus operators based on fractal geometry concepts have garnered attention [21]. Specifically, the Hausdorff derivative [22] and the discrete form thereof, the Hausdorff difference, provide an efficient solution [23]. The Hausdorff difference degrades to the classical integer-order difference when the order is unity, whereas a tunable parameter is introduced at noninteger orders. This property allows the weight of historical information in the current update to be adjusted flexibly. Compared to traditional fractional derivatives [24], the Hausdorff difference is more compact in form and easier to implement discretely. It enables the embedding of dynamic characteristics similar to fractal memory into the update process without increasing computational overhead significantly. The integer-order methods rely on a fixed time scale that is ill-suited for fractal landscapes, whereas traditional fractional algorithms incur prohibitive computational costs due to full history accumulation. This work addresses this dichotomy by introducing a recursive Hausdorff framework that achieves fractal adaptivity while maintaining constant time complexity.

Based on the aforementioned analysis, a Hausdorff difference-based Adam algorithm (HAdam) is proposed in this paper for CNN training by embedding the Hausdorff difference operator into the adaptive moment estimation framework. The momentum update mechanism based on integer-order differences in traditional Adam is reconstructed. The order parameter $\gamma$ is introduced to modulate the momentum term explicitly, as well as the first-order and second-order moment estimates. Theoretical derivation indicates that the order $\gamma$ exerts a significant nonlinear modulation on momentum intensity and the effective learning rate. In the early stage of training, a larger order helps to reinforce the driving role of the current gradient, thereby accelerating convergence. In the later stage, a smaller order helps to smooth the update trajectory, suppress oscillations, and improve the precision of the solution. Furthermore, two adaptive strategies for order adjustment are designed: a nonlinear scheduling based on the iterative process and a dynamic adjustment based on the gradient norm. These strategies enable $\gamma$ to evolve automatically according to the optimization stage and gradient scale, forming a dynamic parameter update mechanism that balances convergence speed and precision. The primary research content and innovative contributions of this paper are summarized as follows. First, the Hausdorff difference formula that is suitable for discrete optimization processes is derived based on Hausdorff derivative theory. This formula is seamlessly integrated with the Adam framework to propose the HAdam optimization algorithm. Second, to address the limitations of fixed orders, adaptive adjustment strategies based on iteration cycles and gradient feedback are constructed. These strategies achieve joint dynamic control of momentum intensity and the effective learning rate. Third, through theoretical analysis and numerical simulation, the influence of the order parameter and the value range thereof on algorithm stability, weight evolution trajectories, and convergence speed is revealed. Principles for selecting orders during different training stages are subsequently proposed. Fourth, experimental verification on standard image classification datasets and typical convolutional network architectures demonstrates that the HAdam algorithm outperforms standard Adam and other related algorithms in terms of convergence efficiency and recognition accuracy.

The remainder of this paper is organized as follows. Section 2 briefly reviews typical optimization algorithms for convolutional neural networks and presents the mathematical definitions of the Hausdorff derivative and difference, clarifying the connection to traditional operators. Section 3 elaborates on the derivation of the HAdam algorithm, provides the parameter update formulas containing the order parameter, and introduces the two adaptive order adjustment strategies. Section 4 presents experimental results on multiple image datasets, contrasting the convergence characteristics and recognition performance of the algorithm under different order settings and adjustment strategies. Finally, Section 5 summarizes the work of this paper and discusses the potential for applying this method to other deep learning architectures and future research directions.

2. Preliminaries

To demonstrate the effectiveness of fractal differences in improving the optimization performance of CNNs, this subsection first introduces a series of optimization algorithms for network parameter training. It then recalls the definition of the Hausdorff derivative.

2.1. Adam Algorithm

$$w_{ij}^{\left(l\right)}\left(k\right)=w_{ij}^{\left(l\right)}(k-1)-\eta g_{ij}^{\left(l\right)}(k-1),$$

(1)

$$b_j^{\left(l\right)}\left(k\right)=b_j^{\left(l\right)}(k-1)-\eta h_j^{\left(l\right)}(k-1).$$

(2)

In the context of the optimization process at iteration k, the weight parameter located at the i-th spatial position within the j-th kernel of layer l is symbolized by $w_{ij}^{\left(l\right)}\left(k\right)$, while its associated bias is denoted by $b_j^{\left(l\right)}\left(k\right)$. The magnitude of the parameter update step is regulated by the learning rate scalar $\eta$. Moreover, with $J\left(k\right)$ representing the objective function value, the terms $g_{ij}^{\left(l\right)}\left(k\right)$ and $h_j^{\left(l\right)}\left(k\right)$ are defined as the partial derivatives of $J\left(k\right)$ with respect to the weights and biases, respectively. The gradients $g_{ij}^{\left(l\right)}\left(k\right)=\partial J\left(k\right)/\partial w_{ij}^{\left(l\right)}\left(k\right)$ and $h_j^{\left(l\right)}\left(k\right)=\partial J\left(k\right)/\partial b_j^{\left(l\right)}\left(k\right)$ are obtained via backpropagation.

In terms of the optimization strategy, the Adam algorithm exhibits properties that are superior to those of traditional methods. Its main feature is an adaptive learning rate scaling mechanism. The effective learning rate at each iteration is adjusted dynamically according to moment estimates of the gradient and is restricted to a reasonable bounded range. By preventing extreme values of the step size, this mechanism significantly improves the numerical stability of the training process.

$$m_{ij}^{\left(l\right)}\left(k\right)={\beta }_1{m}_{ij}^{\left(l\right)}(k-1)+(1-{\beta }_1)g_{ij}^{\left(l\right)}(k-1),$$

(3)

$$v_{ij}^{\left(l\right)}\left(k\right)={\beta }_2{v}_{ij}^{\left(l\right)}(k-1)+(1-{\beta }_2){\left(g_{ij}^{\left(l\right)}\right)}^2(k-1),$$

(4)

$$n_j^{\left(l\right)}\left(k\right)={\beta }_1{n}_j^{\left(l\right)}(k-1)+(1-{\beta }_1)h_j^{\left(l\right)}(k-1),$$

(5)

$$u_j^{\left(l\right)}\left(k\right)={\beta }_2{u}_j^{\left(l\right)}(k-1)+(1-{\beta }_2){\left(h_j^{\left(l\right)}\right)}^2(k-1),$$

(6)

$$\begin{array}{cc}\hfill w_{ij}^{\left(l\right)}\left(k\right)=& w_{ij}^{\left(l\right)}(k-1)-\eta {\displaystyle \frac{{\widehat{m}}_{ij}^{\left(l\right)}\left(k\right)}{\sqrt{{\widehat{v}}_{ij}^{\left(l\right)}\left(k\right)}+\epsilon }},\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill b_j^{\left(l\right)}\left(k\right)=& b_j^{\left(l\right)}(k-1)-\eta {\displaystyle \frac{{\widehat{n}}_j^{\left(l\right)}\left(k\right)}{\sqrt{{\widehat{u}}_j^{\left(l\right)}\left(k\right)}+\epsilon }},\hfill \end{array}$$

(8)

where

$${\widehat{m}}_{ij}^{\left(l\right)}\left(k\right)={\displaystyle \frac{m_{ij}^{\left(l\right)}\left(k\right)}{1-{\beta }_1^k}},{\widehat{n}}_j^{\left(l\right)}\left(k\right)={\displaystyle \frac{n_j^{\left(l\right)}\left(k\right)}{1-{\beta }_1^k}},$$

$${\widehat{v}}_{ij}^{\left(l\right)}\left(k\right)={\displaystyle \frac{v_{ij}^{\left(l\right)}\left(k\right)}{1-{\beta }_2^k}},{\widehat{u}}_j^{\left(l\right)}\left(k\right)={\displaystyle \frac{u_j^{\left(l\right)}\left(k\right)}{1-{\beta }_2^k}}.$$

In this update rule, $\epsilon$ is defined as a small positive real constant to maintain numerical stability, while the scalars ${\beta }_1$ and ${\beta }_2$ serve as critical momentum factors. To capture the dynamic characteristics of parameter evolution, the variables $m_{ij}^{\left(l\right)}\left(k\right)$ and $v_{ij}^{\left(l\right)}\left(k\right)$ accumulate the momentum information for the weights at spatial position i of the j-th convolution kernel in layer l. Correspondingly, $n_j^{\left(l\right)}\left(k\right)$ and $u_j^{\left(l\right)}\left(k\right)$ represent the momentum statistics for the bias parameters within the same kernel and layer. Finally, the k-th powers of ${\beta }_1$ and ${\beta }_2$ are utilized to perform bias correction on the moment estimates, thereby fully constructing the adaptive update mechanism within the Adam-type optimization framework.

2.2. Definition and Discrete Form of the Hausdorff Derivative

For a differentiable function $f\left(t\right)$ defined on the positive real line ${\mathbb{R}}^{+}$ and for an order parameter $\gamma \in {\mathbb{R}}^{+}$, the Hausdorff derivative ${}^{H}D_t^{\gamma }f\left(t\right)$ is defined by

$${}^{H}D_t^{\gamma }f\left(t\right)=\underset{t_0\to t}{lim}{\displaystyle \frac{f\left(t\right)-f\left(t_0\right)}{t^{\gamma }-t_0^{\gamma }}}.$$

(9)

From this definition, a more convenient equivalent expression can be obtained. Specifically, assume that f is continuously differentiable on $(0,\infty )$, $f\in C^1(0,\infty )$, and let $\gamma >0$. Then, for any $t>0$, Equation (9) admits an equivalent differential form by introducing the time-scale variable $\tau =t^{\gamma }$. By using $\tau =t^{\gamma }$ and the chain rule, Equation (9) can be rewritten in the form

$${}^{H}D_t^{\gamma }f\left(t\right)={\displaystyle \frac{1}{\gamma }}{\displaystyle \frac{\mathrm{d}f\left(t\right)}{\mathrm{d}t}}{t}^{1-\gamma }.$$

(10)

In this sense, the Hausdorff derivative can be viewed as a weighted derivative with respect to the transformed time scale $\tau =t^{\gamma }$.

The rationale for employing the Hausdorff derivative lies in its inherent suitability for the fractal and rough loss landscapes typical of deep networks. Unlike standard derivatives that assume a smooth optimization surface, the Hausdorff operator utilizes a fractal metric that adapts to non-uniform geometries. This feature allows the optimizer to navigate complex non-convex terrains more effectively, resulting in superior stability and convergence compared to traditional integer-order methods.

To adapt this concept to a discrete optimization setting, the notion of the Hausdorff derivative is extended to the discrete domain. According to Equation (10), the Hausdorff difference operator for a sequence $f\left(k\right)$ is defined.

To adapt this concept to a discrete optimization setting, the notion of the Hausdorff derivative is extended to the discrete domain. Here, we assume a unit time step $\Delta t=1$ for the discretization process, treating the iteration index k as the discrete equivalent of physical time t. This assumption simplifies the computational complexity while retaining the fractal memory characteristics. According to Equation (10), the Hausdorff difference operator for a sequence $f\left(k\right)$ is defined as

$${}^H▵_k^{\gamma }f\left(k\right)={\displaystyle \frac{1}{\gamma }}\left(f\left(k\right)-f(k-1)\right)k^{1-\gamma }.$$

(11)

It is worth noting that, since k represents the cumulative iteration count, the scale factor $k^{1-\gamma }$ inherently depends on the number of iterations per epoch. Consequently, the memory length of the operator is implicitly influenced by the batch size setting used during training. The Hausdorff difference provides a generalized extension of the classical difference operator. When the order is set to $\gamma =1$, Equation (11) reduces to the standard integer-order first difference,

$${}^H▵_k^{1}f\left(k\right)=f\left(k\right)-f(k-1).$$

In this theoretical framework, the order is defined on the interval $(0,+\infty )$. This parameter determines the degree of deviation of the generalized operator from the classical integer-order case. However, it is important to note that the practical application of the HAdam algorithm imposes further restrictions on $\gamma$ to guarantee numerical stability. The specific algorithmic constraints and the derivation of the effective upper bound for $\gamma$ are detailed in Remark 1. This parameter determines the degree of deviation of the generalized operator from the classical integer-order case. In particular, when the Hausdorff difference is combined with the Adam optimizer with momentum, the parameter $\gamma$ can be used to adjust the effective order of the convolutional neural network and to control the influence of momentum during parameter training. This allows flexible regulation of the optimization process.

3. Construction of Parameter Update Rules via Hausdorff Difference

The Hausdorff difference does not replace the stochastic gradient but functions as a dynamic weighting mechanism within the momentum accumulation process. Specifically, the fractal scaling term $k^{1-\gamma }$ acts as a dual regulator of the effective learning rate and inertia, allowing for nonlinear adaptation of the update trajectory.

Adam Algorithm with Hausdorff Difference

To transform the learning rule with integer-order momentum into its counterpart based on the Hausdorff difference, Equations (3)–(6) are rewritten as follows:

$$\begin{array}{cc}\hfill {}^H▵_k^1{m}_{ij}^{\left(l\right)}\left(k\right)& =m_{ij}^{(l)}(k)-m_{ij}^{(l)}(k-1)\hfill \\ & =({\beta }_1-1)m_{ij}^{(l)}(k-1)+(1-{\beta }_1)g_{ij}^{(l)}(k-1),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill {}^H▵_k^1{v}_{ij}^{(l)}(k)& =v_{ij}^{(l)}(k)-v_{ij}^{(l)}(k-1)\hfill \\ & =({\beta }_2-1)v_{ij}^{(l)}(k-1)+(1-{\beta }_2){(g_{ij}^{(l)})}^2(k-1),\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {}^H▵_k^1{n}_j^{(l)}(k)& =n_j^{(l)}(k)-n_j^{(l)}(k-1)\hfill \\ & =({\beta }_1-1)n_j^{(l)}(k-1)+(1-{\beta }_1)h_j^{(l)}(k-1),\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {}^H▵_k^1{u}_j^{(l)}(k)& =u_j^{(l)}(k)-u_j^{(l)}(k-1)\hfill \\ & =({\beta }_2-1)u_j^{(l)}(k-1)+(1-{\beta }_2){(h_j^{(l)})}^2(k-1).\hfill \end{array}$$

(15)

This modification generalizes the original update rule from the integer-order case to the $\gamma$-order case. By replacing the first-order difference in Equations (12)–(15) with the $\gamma$-order difference, we obtain

$$\begin{array}{cc}\hfill {}^H▵_k^{\gamma }{m}_{ij}^{(l)}(k)& =m_{ij}^{(l)}(k)-m_{ij}^{(l)}(k-1)\hfill \\ & =({\beta }_1-1)m_{ij}^{(l)}(k-1)+(1-{\beta }_1)g_{ij}^{(l)}(k-1),\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill {}^H▵_k^{\gamma }{v}_{ij}^{(l)}(k)& =v_{ij}^{(l)}(k)-v_{ij}^{(l)}(k-1)\hfill \\ & =({\beta }_2-1)v_{ij}^{(l)}(k-1)+(1-{\beta }_2){(g_{ij}^{(l)})}^2(k-1),\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {}^H▵_k^{\gamma }{n}_j^{(l)}(k)& =n_j^{(l)}(k)-n_j^{(l)}(k-1)\hfill \\ & =({\beta }_1-1)n_j^{(l)}(k-1)+(1-{\beta }_1)h_j^{(l)}(k-1),\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill {}^H▵_k^{\gamma }{u}_j^{(l)}(k)& =u_j^{(l)}(k)-u_j^{(l)}(k-1)\hfill \\ & =({\beta }_2-1)u_j^{(l)}(k-1)+(1-{\beta }_2){(h_j^{(l)})}^2(k-1).\hfill \end{array}$$

(19)

In the following, we update the parameter formulas using the Hausdorff difference. Using the Hausdorff difference, Equations (16)–(19) can be written as follows:

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{\gamma }}(m_{ij}^{(l)}(k)-m_{ij}^{(l)}(k-1))k^{1-\gamma }=({\beta }_1-1)m_{ij}^{(l)}(k-1)+(1-{\beta }_1)g_{ij}^{(l)}(k-1),\end{array}$$

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{\gamma }}(v_{ij}^{(l)}(k)-v_{ij}^{(l)}(k-1))k^{1-\gamma }=({\beta }_2-1)v_{ij}^{(l)}(k-1)+(1-{\beta }_2){(g_{ij}^{(l)})}^2(k-1),\end{array}$$

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{\gamma }}(n_j^{(l)}(k)-n_j^{(l)}(k-1))k^{1-\gamma }=({\beta }_1-1)n_j^{(l)}(k-1)+(1-{\beta }_1)h_j^{(l)}(k-1),\end{array}$$

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{\gamma }}(u_j^{(l)}(k)-u_j^{(l)}(k-1))k^{1-\gamma }=({\beta }_2-1)u_j^{(l)}(k-1)+(1-{\beta }_2){(h_j^{(l)})}^2(k-1).\end{array}$$

Accordingly, we obtain the following expressions. These expressions will be used in the subsequent analysis:

$$\begin{array}{c}\hfill m_{ij}^{(l)}(k)=[1+{\displaystyle \frac{\gamma }{k^{1-\gamma }}}({\beta }_1-1)]m_{ij}^{(l)}(k-1)+{\displaystyle \frac{(1-{\beta }_1)\gamma }{k^{1-\gamma }}}{g}_{ij}^{(l)}(k-1),\end{array}$$

(20)

$$\begin{array}{c}\hfill v_{ij}^{(l)}(k)=[1+{\displaystyle \frac{\gamma }{k^{1-\gamma }}}({\beta }_2-1)]v_{ij}^{(l)}(k-1)+{\displaystyle \frac{(1-{\beta }_2)\gamma }{k^{1-\gamma }}}{(g_{ij}^{(l)})}^2(k-1),\end{array}$$

(21)

$$\begin{array}{c}\hfill n_j^{(l)}(k)=[1+{\displaystyle \frac{\gamma }{k^{1-\gamma }}}({\beta }_1-1)]n_j^{(l)}(k-1)+{\displaystyle \frac{(1-{\beta }_1)\gamma }{k^{1-\gamma }}}{h}_j^{(l)}(k-1),\end{array}$$

(22)

$$\begin{array}{c}\hfill u_j^{(l)}(k)=[1+{\displaystyle \frac{\gamma }{k^{1-\gamma }}}({\beta }_2-1)]u_j^{(l)}(k-1)+{\displaystyle \frac{(1-{\beta }_2)\gamma }{k^{1-\gamma }}}{(h_j^{(l)})}^2(k-1).\end{array}$$

(23)

To ensure the algorithm is well-defined at the first iteration ($k=1$), we address the interpretation of the term $k^{1-\gamma }$ and the initial states. At $k=1$, the scaling factor becomes $1^{1-\gamma }=1$, which avoids any numerical singularity in the coefficients of Equations (20)–(23). The variables with index $k-1=0$ refer to the initialization states. Specifically, $g_{ij}^{\left(l\right)}\left(0\right)$ represents the gradient computed at the initial weights $w_{ij}^{\left(l\right)}\left(0\right)$, while the moment estimates are explicitly initialized as $m_{ij}^{\left(l\right)}\left(0\right)=0$ and $v_{ij}^{\left(l\right)}\left(0\right)=0$. Substituting these into Equation (20), the first update step simplifies to $m_{ij}^{\left(l\right)}\left(1\right)=(1-{\beta }_1)\gamma g_{ij}^{\left(l\right)}\left(0\right)$, ensuring a deterministic start to the optimization process.

Equation (23) constitutes the central update mechanism of the proposed algorithm, effectively translating the theoretical properties of the Hausdorff difference into practical parameter adjustments. By dynamically incorporating the order $\gamma$, this mechanism modulates the balance between the retention of historical information and the sensitivity to current gradients. Consequently, the optimizer acts as a dynamic system with a variable time scale, which enhances convergence speed during the transient phases and maintains stability throughout the steady-state training process.

In the HAdam algorithm, the first-order and second-order moment estimates may suffer from initialization deviations. To correct these deviations, the corresponding moment values must be properly initialized. Therefore, we study the solution of Equation (20) as follows.

Theorem 1.

Let $G_{(1,0)}=1$; the solution of Equation (20) can be determined as follows:

$$\begin{array}{c}\hfill m_{ij}^{\left(l\right)}\left(k\right)=G_{(1,k)}{m}_{ij}^{\left(l\right)}\left(0\right)+{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}{g}_{ij}^{\left(l\right)}(k-q),\end{array}$$

(24)

where $G_{(1,k)}={\mu }_{(0,{\beta }_1,k)}{G}_{(1,k-1)}$ and $H_{(1,k)}={\mu }_{(1,{\beta }_1,k)}$.

Proof. 

This theorem is proved by the mathematical induction.

For $k=2$, we have

$$\begin{array}{cc}\hfill m_{ij}^{\left(l\right)}\left(2\right)=& {\mu }_{(0,{\beta }_1,2)}{\mu }_{(0,{\beta }_1,1)}{m}_{ij}^{\left(l\right)}\left(0\right)+{\mu }_{(0,{\beta }_1,2)}{\mu }_{(1,{\beta }_1,1)}{g}_{ij}^{\left(l\right)}\left(0\right)+{\mu }_{(1,{\beta }_1,2)}{g}_{ij}^{\left(l\right)}\left(1\right)\hfill \\ \hfill =& G_{(1,2)}{m}_{ij}^{\left(l\right)}\left(0\right)+{\displaystyle \sum _{q=1}^2}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}{g}_{ij}^{\left(l\right)}(k-q).\hfill \end{array}$$

(25)

Based on Equation (24), we also obtain

$$\begin{array}{cc}\hfill m_{ij}^{\left(l\right)}(k+1)=& {\mu }_{(0,{\beta }_1,k+1)}{m}_{ij}^{\left(l\right)}\left(k\right)+{\mu }_{(1,{\beta }_1,k+1)g_{ij}^{\left(l\right)}\left(k\right)}\hfill \\ \hfill =& {\mu }_{(0,{\beta }_1,k+1)}[G_{(1,k)}{m}_{ij}^{\left(l\right)}\left(0\right)+{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}{g}_{ij}^{\left(l\right)}(k-q)]\hfill \\ & -{\mu }_{(1,{\beta }_1,k+1)g_{ij}^{\left(l\right)}\left(k\right)}\hfill \\ \hfill =& {\mu }_{(0,{\beta }_1,k+1)}{G}_{(1,k)}{m}_{ij}^{\left(l\right)}\left(0\right)-{\mu }_{(0,{\beta }_1,k+1)}{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}{g}_{ij}^{\left(l\right)}(k-q)\hfill \\ & +{\mu }_{(1,{\beta }_1,k+1)g_{ij}^{\left(l\right)}\left(k\right)}\hfill \\ \hfill =& G_{(1,k+1)}{m}_{ij}^{\left(l\right)}\left(0\right)+{\displaystyle \sum _{q=1}^{k+1}}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}{g}_{ij}^{\left(l\right)}(k-q+1).\hfill \end{array}$$

(26)

   □

Similarly, the solutions of Equations (21)–(23) are given as

$$\begin{array}{cc}\hfill v_{ij}^{\left(l\right)}\left(k\right)=& G_{(2,k)}{v}_{ij}^{\left(l\right)}\left(0\right)+{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(2,k)}}{G_{(2,k-q+1)}}}{H}_{(2,k-q+1)}{(g_{ij}^{\left(l\right)})}^2(k-q),\hfill \end{array}$$

$$\begin{array}{cc}\hfill n_j^{\left(l\right)}\left(k\right)=& G_{(1,k)}{n}_j^{\left(l\right)}\left(0\right)+{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}{h}_j^{\left(l\right)}(k-q),\hfill \end{array}$$

$$\begin{array}{cc}\hfill u_j^{\left(l\right)}\left(k\right)=& G_{(2,k)}{u}_j^{\left(l\right)}\left(0\right)+{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(2,k)}}{G_{(2,k-q+1)}}}{H}_{(2,k-q+1)}{(h_j^{\left(l\right)})}^2(k-q),\hfill \end{array}$$

where $G_{(2,k)}={\mu }_{(0,{\beta }_2,k)}{G}_{(2,k-1)}$ and $H_{(2,k)}={\mu }_{(1,{\beta }_2,k)}$.

In the initial iterations of the HAdam algorithm, the first-order and second-order moment estimates depend on historical information, so their values are easily influenced by the initial state and may become biased. To reduce this initialization bias, the corresponding moment variables are initialized to zero before training begins. Specifically, if the initial conditions at iteration 0 are set as $m_{ij}^{\left(l\right)}\left(0\right)=0$, $v_{ij}^{\left(l\right)}\left(0\right)=0$, $n_j^{\left(l\right)}\left(0\right)=0$, and $u_j^{\left(l\right)}\left(0\right)=0$, then the following relations can be derived:

$$\begin{array}{c}\hfill m_{ij}^{\left(l\right)}\left(k\right)={\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}{g}_{ij}^{\left(l\right)}(k-q),\end{array}$$

(27)

$$\begin{array}{c}\hfill v_{ij}^{\left(l\right)}\left(k\right)={\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(2,k)}}{G_{(2,k-q+1)}}}{H}_{(2,k-q+1)}{(g_{ij}^{\left(l\right)})}^2(k-q),\end{array}$$

(28)

$$\begin{array}{c}\hfill n_j^{\left(l\right)}\left(k\right)={\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}{h}_j^{\left(l\right)}(k-q),\end{array}$$

(29)

$$\begin{array}{c}\hfill u_j^{\left(l\right)}\left(k\right)={\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(2,k)}}{G_{(2,k-q+1)}}}{H}_{(2,k-q+1)}{(h_j^{\left(l\right)})}^2(k-q).\end{array}$$

(30)

Then, the mathematical expectations from Equations (27)–(30) are determined by

$$\begin{array}{cc}\hfill E[m_{ij}^{\left(l\right)}\left(k\right)]=& E[{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}{g}_{ij}^{\left(l\right)}(k-q)]\hfill \\ \hfill =& E[g_{ij}^{\left(l\right)}(k-1)]{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}+{\delta }_1,\hfill \\ \hfill E[v_{ij}^{\left(l\right)}\left(k\right)]=& E[{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(2,k)}}{G_{(2,k-q+1)}}}{H}_{(2,k-q+1)}{(g_{ij}^{\left(l\right)})}^2(k-q)]\hfill \\ \hfill =& E[{(g_{ij}^{\left(l\right)})}^2(k-1)]{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(2,k)}}{G_{(2,k-q+1)}}}{H}_{(2,k-q+1)}+{\delta }_2,\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill E[n_j^{\left(l\right)}\left(k\right)]=& E[{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}{h}_j^{\left(l\right)}(k-q)]\hfill \\ \hfill =& E[h_j^{\left(l\right)}(k-1)]{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}+{\delta }_3,\hfill \\ \hfill E[u_j^{\left(l\right)}\left(k\right)]=& E[{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(2,k)}}{G_{(2,k-q+1)}}}{H}_{(2,k-q+1)}{(h_j^{\left(l\right)})}^2(k-q)]\hfill \\ \hfill =& E[{(h_j^{\left(l\right)})}^2(k-1)]{\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(2,k)}}{G_{(2,k-q+1)}}}{H}_{(2,k-q+1)}+{\delta }_4,\hfill \end{array}$$

(32)

Here, ${\delta }_r$ with $r=1,2,3,4$ represents the residual term caused by the non-stationarity of the gradients during the update process. Strictly defined, ${\delta }_r$ accounts for the difference between the expectation of historical gradients and the expectation of the current gradient. For instance, ${\delta }_1$ is defined as

$${\delta }_1={\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}\left(E[g_{ij}^{\left(l\right)}(k-q)]-E[g_{ij}^{\left(l\right)}(k-1)]\right).$$

Assuming the objective function is smooth and the learning rate is sufficiently small, the change in the gradient expectation between adjacent steps is minimal. Consequently, ${\delta }_r$ approaches zero, validating the approximation in the moment estimation. Similar definitions apply to ${\delta }_2$, ${\delta }_3$, and ${\delta }_4$.

Based on the zero-initialization condition $m\left(0\right)=0$, a systematic bias exists in the early training stages. Unlike standard Adam, where the bias decay is constant, HAdam’s bias structure is time-varying and depends on $\gamma$. To strictly eliminate this specific initialization bias, we do not use the standard correction factor. Instead, we derive the exact correction terms based on the coefficient expansion in Theorem 1. The corrected moment estimation formulas are defined as follows:

$$\begin{array}{cc}\hfill & {\widehat{m}}_{ij}^{\left(l\right)}\left(k\right)={\displaystyle \frac{m_{ij}^{\left(l\right)}\left(k\right)}{{\displaystyle \sum _{q=1}^k}\frac{G_{(1,k)}}{G_{(1,k-q+1)}}{H}_{(1,k-q+1)}}},\hfill \\ \hfill & {\widehat{v}}_{ij}^{\left(l\right)}\left(k\right)={\displaystyle \frac{v_{ij}^{\left(l\right)}\left(k\right)}{{\displaystyle \sum _{q=1}^k}\frac{G_{(2,k)}}{G_{(2,k-q+1)}}{H}_{(2,k-q+1)}}},\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill & {\widehat{n}}_j^{\left(l\right)}\left(k\right)={\displaystyle \frac{n_{ij}^{\left(l\right)}\left(k\right)}{{\displaystyle \sum _{q=1}^k}\frac{G_{(1,k)}}{G_{(1,k-q+1)}}{H}_{(1,k-q+1)}}},\hfill \\ \hfill & {\widehat{u}}_j^{\left(l\right)}\left(k\right)={\displaystyle \frac{u_{ij}^{\left(l\right)}\left(k\right)}{{\displaystyle \sum _{q=1}^k}\frac{G_{(2,k)}}{G_{(2,k-q+1)}}{H}_{(2,k-q+1)}}}.\hfill \end{array}$$

(34)

To prevent the computational complexity from increasing linearly with iterations, we implement the bias correction terms recursively. Let ${\mathrm{\Omega }}_m\left(k\right)$ and ${\mathrm{\Omega }}_v\left(k\right)$ denote the denominators for the first-order and second-order moment corrections in Equations (33) and (34), respectively. Taking ${\mathrm{\Omega }}_m\left(k\right)$ as an example,

$${\mathrm{\Omega }}_m\left(k\right)={\displaystyle \sum _{q=1}^k}{\displaystyle \frac{G_{(1,k)}}{G_{(1,k-q+1)}}}{H}_{(1,k-q+1)}.$$

(35)

Using the property $G_{(1,k)}={\mu }_{(0,{\beta }_1,k)}{G}_{(1,k-1)}$, this term can be rewritten as a recurrence relation:

$${\mathrm{\Omega }}_m\left(k\right)={\mu }_{(0,{\beta }_1,k)}{\mathrm{\Omega }}_m(k-1)+{\mu }_{(1,{\beta }_1,k)},$$

(36)

with the initial condition ${\mathrm{\Omega }}_m\left(0\right)=0$. A similar recursive form applies to ${\mathrm{\Omega }}_v\left(k\right)$. This recursive implementation ensures that the bias correction step has a constant time complexity of $O\left(1\right)$ per iteration, making HAdam computationally efficient and suitable for large-scale training.

To eliminate the estimation bias generated in the initialization stage, the corrected moment estimates, such as ${\widehat{m}}_{ij}^{\left(l\right)}\left(k\right)$, are first computed by using Equations (33) and (34). These corrected terms are then applied in the HAdam optimization framework to determine the final parameter update equations with order-dependent momentum.

$$\begin{array}{cc}\hfill & w_{ij}^{\left(l\right)}\left(k\right)=w_{ij}^{\left(l\right)}(k-1)-\eta {\displaystyle \frac{{\widehat{m}}_{ij}^{\left(l\right)}\left(k\right)}{\sqrt{{\widehat{v}}_{ij}^{\left(l\right)}\left(k\right)}+\epsilon }},\hfill \\ \hfill & b_j^{\left(l\right)}\left(k\right)=b_j^{\left(l\right)}(k-1)-\eta {\displaystyle \frac{{\widehat{n}}_j^{\left(l\right)}\left(k\right)}{\sqrt{{\widehat{u}}_j^{\left(l\right)}\left(k\right)}+\epsilon }}.\hfill \end{array}$$

(37)

To study the role of the order parameter in the HAdam update, we introduce the following coefficients (Algorithm 1). If we set ${\mu }_{(0,{\beta }_1,k)}=[1+\gamma ({\beta }_1-1)/k^{1-\gamma }]$, ${\mu }_{(1,{\beta }_1,k)}=(1-{\beta }_1)\gamma /k^{1-\gamma }$, ${\mu }_{(0,{\beta }_2,k)}=[1+\gamma ({\beta }_2-1)/k^{1-\gamma }]$, and ${\mu }_{(1,{\beta }_2,k)}=(1-{\beta }_2)\gamma /k^{1-\gamma }$, then we can analyze how the order $\gamma$ affects the weight update formulas. The influence of the order on the weight update formulas is examined in the following cases. Letting ${\beta }_1=0.9$ and ${\beta }_2=0.999$, we select the order as $\gamma =0.2,0.4,\dots ,0.8$ and $\gamma =1.05,1.1,1.2,1.4$, respectively, for the cases that are $\gamma \in (0,1]$ and $\gamma \in (1,+\infty )$. Letting ${\beta }_1=0.9$ and ${\beta }_2=0.999$, we select the order as $\gamma =0.2,0.4,\dots ,0.8$ and $\gamma =1.05,1.1,1.2,1.4$, respectively, for the cases that are $\gamma \in (0,1]$ and $\gamma \in (1,+\infty )$. The curves of ${\mu }_{(0,{\beta }_1,k)}$ with different orders $\gamma =0.2,0.4,\dots ,0.8$ and $\gamma =1.05,1.1,1.2,1.4$ are drawn in Figure 1.

Algorithm 1 HAdam optimization algorithm with recursive bias correction

Input: training set $D=\left\{(x_{i,j}^{\left(l\right)},y_{i,j}^{\left(l\right)})\right\}$, learning rate $\eta$, exponential decay rates ${\beta }_1,{\beta }_2\in [0,1)$, order parameter $\gamma >0$, objective function J, small constant $\epsilon$. Initialize: $m_{i,j}^{\left(l\right)}\left(0\right)\leftarrow 0$, $v_{i,j}^{\left(l\right)}\left(0\right)\leftarrow 0$, $n_j^{\left(l\right)}\left(0\right)\leftarrow 0$, $u_j^{\left(l\right)}\left(0\right)\leftarrow 0$. Initialize bias correction terms: ${\mathrm{\Omega }}_m\left(0\right)\leftarrow 0$, ${\mathrm{\Omega }}_v\left(0\right)\leftarrow 0$. for $k=1,2,\dots ,K$ do   Step 1: Update raw moment estimates $$\begin{array}{cc}\hfill m_{ij}^{\left(l\right)}\left(k\right)& =\left[1+{\displaystyle \frac{\gamma }{k^{1-\gamma }}}({\beta }_1-1)\right]m_{ij}^{\left(l\right)}(k-1)+{\displaystyle \frac{(1-{\beta }_1)\gamma }{k^{1-\gamma }}}{g}_{ij}^{\left(l\right)}(k-1),\hfill \\ \hfill v_{ij}^{\left(l\right)}\left(k\right)& =\left[1+{\displaystyle \frac{\gamma }{k^{1-\gamma }}}({\beta }_2-1)\right]v_{ij}^{\left(l\right)}(k-1)+{\displaystyle \frac{(1-{\beta }_2)\gamma }{k^{1-\gamma }}}{\left(g_{ij}^{\left(l\right)}(k-1)\right)}^2.\hfill \end{array}$$   Step 2: Update recursive bias correction terms $$\begin{array}{cc}\hfill {\mathrm{\Omega }}_m\left(k\right)& =\left[1+{\displaystyle \frac{\gamma }{k^{1-\gamma }}}({\beta }_1-1)\right]{\mathrm{\Omega }}_m(k-1)+{\displaystyle \frac{(1-{\beta }_1)\gamma }{k^{1-\gamma }}},\hfill \\ \hfill {\mathrm{\Omega }}_v\left(k\right)& =\left[1+{\displaystyle \frac{\gamma }{k^{1-\gamma }}}({\beta }_2-1)\right]{\mathrm{\Omega }}_v(k-1)+{\displaystyle \frac{(1-{\beta }_2)\gamma }{k^{1-\gamma }}}.\hfill \end{array}$$   Step 3: Compute bias-corrected moments $${\widehat{m}}_{ij}^{\left(l\right)}\left(k\right)=m_{ij}^{\left(l\right)}\left(k\right)/{\mathrm{\Omega }}_m\left(k\right),{\widehat{v}}_{ij}^{\left(l\right)}\left(k\right)=v_{ij}^{\left(l\right)}\left(k\right)/{\mathrm{\Omega }}_v\left(k\right).$$   Similarly compute ${\widehat{n}}_j^{\left(l\right)}\left(k\right)$ and ${\widehat{u}}_j^{\left(l\right)}\left(k\right)$ using corresponding $n,u$ and $\mathrm{\Omega }$   Step 4: Update parameters $$\begin{array}{cc}\hfill w_{ij}^{\left(l\right)}\left(k\right)& =w_{ij}^{\left(l\right)}(k-1)-\eta {\displaystyle \frac{{\widehat{m}}_{ij}^{\left(l\right)}\left(k\right)}{\sqrt{{\widehat{v}}_{ij}^{\left(l\right)}\left(k\right)}+\epsilon }},\hfill \\ \hfill b_j^{\left(l\right)}\left(k\right)& =b_j^{\left(l\right)}(k-1)-\eta {\displaystyle \frac{{\widehat{n}}_j^{\left(l\right)}\left(k\right)}{\sqrt{{\widehat{u}}_j^{\left(l\right)}\left(k\right)}+\epsilon }}.\hfill \end{array}$$ end for Output: updated network parameters $w_{ij}^{\left(l\right)}\left(K\right)$, $b_j^{\left(l\right)}\left(K\right)$.

From the curves in Figure 1, we observe that, as k increases, ${\mu }_{(0,{\beta }_1,k)}$ also increases for all $\gamma \in (0,1]$. This indicates that the coefficient ${\mu }_{(0,{\beta }_1,k)}$ grows monotonically with respect to the iteration index k under the considered range of $\gamma$. For $\gamma \in (1,+\infty )$, a decrease in $\gamma$ likewise leads to an increase in ${\mu }_{(0,{\beta }_1,k)}$. However, with an increase in k, ${\mu }_{(0,{\beta }_1,k)}$ gradually decreases, reaching even negative values for all $\gamma$. Therefore, avoiding such conditions is crucial to prevent instability in parameter tuning. To ensure the boundedness and non-negativity of the key coefficients throughout the training process, a strict sufficient condition is required. Specifically, for a maximum iteration count M, the order $\gamma$ must satisfy $\gamma M^{\gamma -1}(1-\beta )\le 1$. A detailed derivation of this stability criterion and the resulting upper bounds for $\gamma$ are provided in Remark 1.

The values of $\gamma$ were selected as $\gamma =0.2,0.4,\dots ,0.8$ for the range $\gamma \in (0,1]$ and as $\gamma =1.05,1.1,1.2,1.4$ for the range $\gamma \in (1,+\infty )$ to examine the behavior of ${\mu }_{(1,{\beta }_1,k)}$ with respect to $\gamma$. Figure 2 shows the curves of ${\mu }_{(1,{\beta }_1,k)}$ for the selected values of $\gamma$. These curves illustrate how the coefficient ${\mu }_{(1,{\beta }_1,k)}$ varies with the iteration index k under different choices of the order $\gamma$.

From Figure 2, we observe that ${\mu }_{(1,{\beta }_1,k)}$ increases as the order $\gamma$ increases. For each order $\gamma \in (0,1]$, a decrease in k results in an increase in ${\mu }_{(1,{\beta }_1,k)}$. Equivalently, for fixed $\gamma$, ${\mu }_{(1,{\beta }_1,k)}$ decreases as k increases, which indicates that the contribution of this coefficient becomes smaller at later iterations. For $\gamma \in (1,+\infty )$, an increase in $\gamma$ likewise leads to an increase in ${\mu }_{(1,{\beta }_1,k)}$. However, contrary to the previous range, an increase in k results in an increase in ${\mu }_{(1,{\beta }_1,k)}$ for all $\gamma$.

From the viewpoint of parameter interpretation, ${\mu }_{(0,{\beta }_1,k)}$ can be regarded as a coefficient that measures the influence of the momentum term, and ${\mu }_{(1,{\beta }_1,k)}$ can be viewed as a coefficient that reflects the effective global learning rate. For $\gamma \in (0,1]$, ${\mu }_{(0,{\beta }_1,k)}$ decreases monotonically as $\gamma$ increases, whereas ${\mu }_{(1,{\beta }_1,k)}$ increases monotonically. This means that a larger order weakens the contribution of the momentum term and strengthens the effect of the learning rate. Hence, in the initial stage of network training, it is desirable that ${\mu }_{(0,{\beta }_1,k)}$ is relatively small and ${\mu }_{(1,{\beta }_1,k)}$ is relatively large so that parameter updates are accelerated. In the later stage of training, it is preferable that ${\mu }_{(0,{\beta }_1,k)}$ is relatively large and ${\mu }_{(1,{\beta }_1,k)}$ is relatively small in order to improve recognition accuracy. When the iteration index k is small, which corresponds to the early training phase, a relatively large-order $\gamma \in (0,1]$ should be chosen to speed up convergence. When k becomes large and the training enters a later phase, a relatively small order $\gamma$ in the same interval should be selected to obtain a more refined optimization result. For $\gamma \in (1,+\infty )$, ${\mu }_{(0,{\beta }_1,k)}$ increases as $\gamma$ increases, while ${\mu }_{(1,{\beta }_1,k)}$ decreases as $\gamma$ increases. Therefore, in this range, a relatively small order $\gamma$ is recommended if a higher weight update speed is required. Overall, the variation of ${\mu }_{(0,{\beta }_1,k)}$ and ${\mu }_{(1,{\beta }_1,k)}$ across different training stages and order intervals provides an intuitive rule for selecting $\gamma$ so as to balance rapid convergence and high recognition accuracy.

During network training, the choice of the order $\gamma$ can be adjusted in coordination with the training stage and the gradient magnitude. In the early stage of training, the gradients of the parameters are relatively large. A larger-order $\gamma$ is then used so that the update step becomes faster and convergence is accelerated. In the later stage of training, the gradients gradually decrease. A smaller-order $\gamma$ is then adopted in order to enhance the smoothness and refinement of the updates and to improve the final recognition accuracy. Accordingly, ${\mu }_{(0,{\beta }_1,k)}$ and ${\mu }_{(1,{\beta }_1,k)}$ play complementary roles at different stages and support the transition from fast approach to fine adjustment.

Based on this idea, the order $\gamma$ can be set adaptively according to the rules given in Equations (38)–(40). In this way, a mechanism is obtained in which the order is adjusted automatically by the iteration progress and the gradient information. Furthermore, two concrete methods for selecting the order are proposed, which provide simple and practical guidance for choosing $\gamma$ in different training scenarios.

Adjustment Method One Nonlinear order adjustment based on a cosine function.

During optimization, in the initial stage of training, a relatively large-order $\gamma$ is selected to accelerate convergence. In the later stage of training, a smaller-order $\gamma$ is preferred in order to improve optimization accuracy. Based on this idea, a cosine function is used in this method to adjust the order $\gamma$ in a nonlinear manner. In the lth layer, the orders ${\gamma }_{ij}^{\left(l\right)}\left(k\right)$ for weight updates and ${\gamma }_j^{\left(l\right)}\left(k\right)$ for bias updates vary smoothly from ${\gamma }_{max}$ to ${\gamma }_{min}$ within the interval $[{\gamma }_{min},{\gamma }_{max}]$ as the iteration index k increases. This strategy implements a natural transition from a stage of accelerated convergence to a stage of fine adjustment. It provides a simple and practical nonlinear scheme for selecting the orders associated with the weights and the biases.

$${\gamma }_{ij}^{\left(l\right)}\left(k\right)=\cos ({\displaystyle \frac{\pi }{2}}·{\displaystyle \frac{k-1}{M-1}})({\gamma }_{max}-{\gamma }_{min})+{\gamma }_{min},$$

(38)

$${\gamma }_j^{\left(l\right)}\left(k\right)=\cos ({\displaystyle \frac{\pi }{2}}·{\displaystyle \frac{k-1}{M-1}})({\gamma }_{max}-{\gamma }_{min})+{\gamma }_{min}.$$

(39)

Adjustment Method Two Nonlinear adjustment based on a hyperbolic tangent function.

This strategy is designed to optimize the order configuration dynamically by using gradient feedback. In the early stage of optimization, large gradient norms require a fast response. In the later stage, small gradient norms require fine adjustment. A hyperbolic tangent function is used to construct the dependence of the order $\gamma$ on the gradient norm. This design allows the order to adapt to the current gradient magnitude through expansion or contraction. In this way, fast descent is promoted in the initial phase, and the quality of convergence is improved in the later phase. A dynamic balance is thus achieved over the whole optimization process.

$${\gamma }_{ij}^{\left(l\right)}\left(k\right)=\tanh \left(\right|g_{ij}^{\left(l\right)}(k-1)\left|\right)({\gamma }_{max}-{\gamma }_{min})+{\gamma }_{min},$$

(40)

$${\gamma }_j^{\left(l\right)}\left(k\right)=\tanh \left(\right|h_j^{\left(l\right)}(k-1)\left|\right)({\gamma }_{max}-{\gamma }_{min})+{\gamma }_{min},$$

(41)

where ${\gamma }_{min}$ and ${\gamma }_{max}$ denote the minimum and maximum values, respectively, in the range of ${\gamma }_{ij}^{\left(l\right)}\left(k\right)$ or ${\gamma }_j^{\left(l\right)}\left(k\right)$. To guarantee the robustness of the proposed method and minimize the necessity for layer-specific manual tuning, ${\gamma }_{min}$ and ${\gamma }_{max}$ are established as global hyperparameters shared across all network layers. The upper bound ${\gamma }_{max}$ is strictly constrained by the theoretical stability criterion derived in Remark 1 (specifically, $\gamma \le \overline{\gamma }$), whereas the lower bound ${\gamma }_{min}$ is assigned a small positive value to exploit the acceleration characteristics inherent to fractional orders. Empirical evidence indicates that the interval $[0.2,1.05]$ consistently yields effective performance across diverse tasks.

Remark 1.

Because ${\mu }_{(0,{\beta }_1,k)}$ may occur for $\gamma \in (1,+\infty )$, we need to take a terminal value of γ. Letting ${\mu }_{(0,{\beta }_1,k)}=[1+\gamma ({\beta }_1-1)/k^{1-\gamma }]$, we get

$${\displaystyle \frac{\gamma }{k^{1-\gamma }}}({\beta }_1-1)>-1.$$

Hence, we have

$$\gamma k^{\gamma -1}({\beta }_1-1)>-1.$$

Due to $k^{\gamma -1}\le M^{\gamma -1}$ with $0<k\le M$, $\mu <1$, and $\gamma >1$, where M is the total number of iterations, we get

$$\gamma k^{\gamma -1}({\beta }_1-1)\ge \gamma M^{\gamma -1}({\beta }_1-1).$$

Letting

$$\gamma M^{\gamma -1}({\beta }_1-1)\le -1,$$

we set order ${\gamma }_0$ to satisfy

$${\gamma }_0{M}^{{\gamma }_0-1}={\displaystyle \frac{1}{1-{\beta }_1}},$$

and then the condition $\gamma \in (0,{\gamma }_0]$ can make ${\mu }_{(0,{\beta }_1,k)}$ not be less than zero. We can set the order ${\gamma }_1$ to satisfy the following equation as

$${\gamma }_1{M}^{{\gamma }_1-1}={\displaystyle \frac{1}{1-{\beta }_1}}.$$

Then, we can use ${\gamma }_1$ to determine the effective interval of γ. For example, let $M=940$, ${\beta }_1=0.9$, and then we get ${\gamma }_1=1.2982$; hence, the interval of γ should be set as $\gamma \in (0,1.2982]$.

Let order ${\gamma }_2$ satisfy the following equation by the same way as

$${\gamma }_2{M}^{{\gamma }_2-1}={\displaystyle \frac{1}{1-{\beta }_2}}.$$

For example, we set $M=940$, ${\beta }_2=0.999$, and then ${\gamma }_2=1.6037$ is determined; then, the effective interval of γ should be set as $\gamma \in (0,1.6037]$.

Then, the effective interval of γ is $\gamma \in (0,\overline{\gamma }]$, where $\overline{\gamma }=min\left({\gamma }_1,{\gamma }_2\right)$. For this example, the effective interval of γ should be set as $\gamma \in (0,1.2982]$ from the above analysis in Adam algorithm with Hausdorff difference.

4. Experimental Results Analysis

In this section, the performance of the proposed algorithms is evaluated on the AlexNet architecture. Two representative datasets are used in the experiments. The first one is the Fashion-MNIST dataset, which contains 70,000 grayscale images of size $28\times 28$ from 10 categories of clothing. Among them, 60,000 images are used for training and 10,000 images are used for testing. This dataset is adopted to examine the behavior of the algorithms on grayscale images. The second one is the CIFAR-10 dataset, which consists of 60,000 color images of size $32\times 32$ with three channels. It also has 10 categories, including common objects such as animals and vehicles. In this case, 50,000 images are used as the training set and 10,000 images are used as the test set. This dataset is used to assess the applicability and robustness of the algorithms on color natural images.

To ensure numerical stability and prevent gradient scaling issues in the tanh-based adjustment strategy, standard input normalization is applied to all datasets. Specifically, pixel values are scaled to the range $[0,1]$ prior to training. This preprocessing step ensures that the gradient magnitudes remain within a dynamic range that allows the tanh function to operate effectively without permanent saturation. For the experiments on the Fashion-MNIST dataset, both the HAdam algorithm and the standard Adam optimizer are configured with an identical learning rate of 0.001 and trained for 100 epochs. For the experiments on the CIFAR-10 dataset, the learning rate for both optimizers is set to 0.0001, and the number of training epochs is set to 200. In both cases, a batch size of 64 is adopted, and the default moment decay rates (${\beta }_1=0.9,{\beta }_2=0.999$) are used. By enforcing consistent hyperparameter settings, we isolate the impact of the proposed Hausdorff difference operator on the optimization performance.

4.1. Influence of the Order $\gamma$ on the Performance of the HAdam Algorithm

To evaluate the impact of the order parameter, the selection of these candidate intervals is grounded in the theoretical analysis presented in Section 3. Specifically, the upper bound of the test range is limited by the stability constraint $\overline{\gamma }$ derived in Remark 1 to prevent coefficient divergence, while the lower bound allows for the exploration of fractional memory effects. Accordingly, we investigate ranges that cover the stable fractional regime $(0,1]$ and the valid super-integer regime $(1,\overline{\gamma }]$.

To determine an appropriate range for the order parameter $\gamma$ in the HAdam algorithm, several candidate intervals are considered, namely $\gamma \in [0.2,0.35]$, $[0.75,0.9]$, $[0.2,1.05]$, $[0.9,1.05]$, $[1.0,1.15]$, and $[1.05,1.2]$. For each interval of $\gamma$, the recognition accuracy and the training loss on the Fashion-MNIST dataset are computed. The HAdam algorithm with Adjustment Method One is compared with the standard Adam algorithm. The corresponding results are summarized in Figure 3, where the performance variation under different order intervals can be observed directly.

Figure 3 shows that, for the HAdam algorithm, recognition accuracy improves and training loss decreases as the order parameter $\gamma$ increases. Across the six tested intervals, the recognition accuracy is highest and the training loss is lowest in the interval $[0.2,1.05]$, whereas the worst results occur in the interval $[0.2,0.35]$ when using the HAdam algorithm. These observations suggest that including a broader range of $\gamma$ values allows the method to better adapt to different training stages. Therefore, a wider interval for the order parameter $\gamma$ is preferable when tuning the parameters of a CNN with the HAdam algorithm. Moreover, for intervals of the same length, a larger-order $\gamma$ consistently yields better performance than a smaller-order $\gamma$ when the HAdam algorithm is applied. Overall, the performance of the HAdam algorithm with Adjustment Method One is satisfactory compared with the Adam algorithm, except in the interval $[0.2,0.35]$.

We also evaluate the performance of the second-order adjustment method that incorporates gradient information for CNN optimization. The same order intervals as in Adjustment Method One are used; that is, several different intervals of $\gamma$ are considered. For each interval, the recognition accuracy and the training loss are computed for the HAdam algorithm with Adjustment Method Two and for the Adam algorithm. The results are shown in Figure 4. These results are used to examine the effect of explicitly introducing gradient information into the order adjustment strategy on the overall performance of the network.

From Figure 4, it can be seen that the results obtained with Adjustment Method Two are similar in overall trend to those obtained with Adjustment Method One. Among the six candidate intervals, when the order interval is $[0.2,1.05]$, the HAdam algorithm with Adjustment Method Two achieves relatively high recognition accuracy and low training loss. In contrast, in the narrower interval $[0.2,0.35]$, the recognition performance of HAdam is the worst and the training loss is the largest. Two observations can be summarized. First, a moderate increase in the length of the order interval tends to improve recognition performance. Second, for a fixed interval length, an increase in the order values usually improves the performance of the algorithm. In general, except for the interval $[0.2,0.35]$, the HAdam algorithm with Adjustment Method Two outperforms the Adam algorithm in CNN optimization for the other suitable order intervals.

On the CIFAR-10 dataset, comparative experiments are carried out for the HAdam algorithm with two nonlinear order adjustment strategies and for the standard Adam algorithm. The corresponding results are presented in Figure 5. This figure reports the recognition accuracy and the training loss of the different algorithms and allows a direct comparison of their optimization performance on CIFAR10.

From Figure 5, it can be seen that the recognition accuracy increases when the length of the interval for the order parameter $\gamma$ becomes larger. For a fixed interval length, an increase in the order $\gamma$ also improves the performance of the HAdam algorithm. These observations indicate that both the length of the interval and the magnitude of the order have an important influence on the optimization effect of HAdam. In general, for all the order intervals except $[0.2,0.35]$, the HAdam algorithm achieves better performance than the Adam algorithm.

The enhanced stability observed during the initial training phase is physically grounded in the metric scaling properties of the Hausdorff operator. Specifically, the term $k^{1-\gamma }$ in the update rule acts as a structural regularizer against the high-variance gradients typically encountered at the start of optimization. Unlike standard Adam, which may suffer from aggressive steps leading to oscillation, the proposed method utilizes this fractal scaling to effectively dampen initial fluctuations. This allows the optimizer to navigate the steep curvature of the loss landscape with reduced risk of divergence, securing a stable trajectory for subsequent acceleration.

The experimental results are averaged over 20 independent runs to statistically mitigate the impact of random initialization and stochastic gradient noise. This sample size is sufficient to ensure that the observed performance trends are statistically significant and consistent. On the Fashion-MNIST dataset, the recognition accuracy of the HAdam algorithm with Adjustment Method One and with Adjustment Method Two as a function of the order $\gamma$ is summarized in

Table 1. Comparison of recognition accuracy for HAdam using different adaptive adjustment strategies.

Method	Interval of Order $\mathit{\gamma }$
	[0.2, 0.35]	[0.75, 0.9]	[0.2, 1.05]	[1.05, 1.2]
Adjustment Method One	0.8859	0.8956	0.9005	0.8998
Adjustment Method Two	0.8860	0.8983	0.9023	0.9006

. This table provides a direct comparison of the combined influence of the order and the two adjustment strategies on recognition accuracy.

By combining the results shown in Figure 4 and Figure 5, several conclusions can be drawn. First, except for the order interval $[0.2,0.35]$, the HAdam algorithm shows overall better performance than the Adam algorithm in CNN optimization. Second, the proposed adaptive order strategies are suitable for two-dimensional image classification tasks and can consistently improve recognition performance. In addition, the overall influence of the order choice on performance is similar for Adjustment Method One and Adjustment Method Two. Based on these observations, it can be considered that selecting a relatively large order range in the interval $(0,\overline{\gamma }]$, together with adaptive tuning of the momentum and the learning rate, is beneficial for improving recognition accuracy and accelerating convergence. These conclusions provide practical guidance for choosing the order $\gamma$ and its interval in actual training.

The superior performance of HAdam is attributed to the fractal time-scale mechanism $k^{1-\gamma }$, which acts as an adaptive regularization coefficient to effectively dampen early-phase gradient oscillations and accelerate subsequent convergence. Furthermore, the recursive formulation of this operator preserves the theoretical benefits of fractional long-term memory while maintaining constant computational complexity, thereby achieving a synergistic balance between efficiency and optimization robustness.

4.2. Overheads and Limitations

While the recursive calculation of dynamic coefficients introduces a marginal scalar overhead, the algorithm maintains a constant time complexity of $O\left(1\right)$ per iteration, yielding wall-clock training times comparable to standard Adam. Regarding limitations, the adaptive order parameter $\gamma$ expands the hyperparameter search space for future research.

5. Conclusions

This study addressed the persistent challenge of optimizing CNNs within complex non-convex loss landscapes, where traditional integer-order optimizers often suffer from slow convergence and entrapment in local minima. To overcome these limitations, we proposed a novel optimizer, HAdam, which incorporates the Hausdorff difference operator to introduce a fractal time-scale mechanism into the gradient update rule. A key innovation of this work is the derivation of a recursive update formulation that reduces the computational complexity of fractional dynamics from linear growth to a constant $O\left(1\right)$, thereby ensuring scalability.

The experimental evaluations on the Fashion-MNIST and CIFAR-10 datasets demonstrated that the HAdam algorithm significantly outperforms state-of-the-art methods. Specifically, the proposed method achieved faster initial convergence and higher final classification accuracy compared to standard Adam and existing fractional-order variants, validating the effectiveness of the adaptive order strategy in balancing exploration and exploitation.

Future research will focus on two specific directions. First, we aim to establish rigorous theoretical convergence bounds for the adaptive order parameter $\gamma$ under non-convex conditions. Second, given the recursive efficiency of the proposed operator, we plan to extend the application of HAdam to the training of Transformer-based architectures and Large Language Models to investigate its potential in processing sequential data with long-term dependencies.