FAdamWav: A Fractional Wavelet Gradient Optimizer for Neural Networks

Abstract

The optimizer is a critical element of neural networks because it computes their optimal parameters through a training process. The Adam optimizer is considered the state of the art in deep learning. However, a drawback is the cost of storing and computing their gradients. A useful tool for addressing this issue is the application of the wavelet transform, and other relevant tool is the fractional derivative, which can be used to create fractional gradient optimizers. This research combines the wavelet transform and fractional optimizers to propose FAdamWav, a fractional version of Adam that uses (i) a parametric discrete wavelet transform to theoretically save $50\%$, $75\%$ or $87.5\%$ of gradient’s memory with one, two or three transformation levels, and (ii) a fractional gradient to optimize the neural network parameters. Experiments indicate that the saved memory is lower than the theoretical bounds, but memory is saved and fractional wavelet-based optimizers have competitive performance compared to their non-fractional and non-wavelet counterparts.

1. Introduction

In the context of artificial neural networks, an important component is the optimizer which can be used in the training process to obtain the optimal parameters of a neural network and extract intrinsic patterns of the input data. Due to the rapid development of Artificial Intelligence, several gradient-based optimizers have been proposed, starting with the fundamental Gradient Descent (GD) and evolving into more sophisticated versions with momentum and learning rate adaptation strategies.

Effectively, GD is the cornerstone for all the gradient-based optimizers given its fundamental principle of updating a parameter ${\theta }_t$ in the opposite direction to the gradient $g_t$. Mathematically, it can be expressed as follows:

$$\Delta {\theta }_t=-\eta g_t$$

(1)

where the learning rate $\eta$ modulates the update for stepping towards getting a minimum [1]. The parameter update Formula (1) considers the entire training dataset and its know as the Vanilla Gradient Descent optimizer (VGD), whereas the Stochastic Gradient Descent (SGD) updates the parameters for each ith training sample [2], and its corresponding formula is:

$$\Delta {\theta }_{t,i}=-\eta g_{t,i}.$$

(2)

Beyond SGD, Adagrad [3] calculates a diagonal matrix $G_{t,ii}$ which accumulates the sum of squares of the historical gradients to modify the adjustment of each parameter ${\theta }_i$. Thus, the Adagrad optimizer equation is:

$$\Delta {\theta }_{t,i}=-{\displaystyle \frac{\eta }{\sqrt{G_{t,ii}+ϵ}}}{g}_{t,i}$$

(3)

where $ϵ>0$ is introduced to avoid the indetermination in case of a zero gradient.

The Adadelta optimizer [4] deals with a disadvantage of Adagrad concerning the loss of the “unit of measure”, given that $\eta$ is adimmensional in Equation (3). Therefore, the numerator $\eta$ can be replaced with the RMS of $\Delta \theta$, up to $t-1$, by considering that:

$$RMS{[\Delta \theta ]}_t=\sqrt{E{[\Delta {\theta }^2]}_t+ϵ}=\sqrt{\gamma E{[\Delta {\theta }^2]}_{t-1}+(1-\gamma )\Delta {\theta }_t^2+ϵ}$$

(4)

whereas the denominator can be replaced with the latest RMS of $g_t$. Hence, the Adadelta formula is:

$$\Delta {\theta }_t=-{\displaystyle \frac{RMS{[\Delta \theta ]}_{t-1}}{RMS{\left[g\right]}_t}}{g}_t$$

(5)

where

$$RMS{\left[g\right]}_t=\sqrt{E{\left[g^2\right]}_t+ϵ.}$$

(6)

Adadelta considers an average window given $0\le \gamma \le 1$ to ponderate the squares of current and previous one gradients according to the following formula:

$$E{\left[g^2\right]}_t=\gamma E{\left[g^2\right]}_{t-1}+(1-\gamma )g_t^2.$$

(7)

The sophisticated Adam optimizer [5] combines the average of past gradients described by:

$$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t$$

(8)

and the past of squared gradients expressed as:

$$v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2$$

(9)

appealing to the momentum concept. The formula for $m_t$ contains a previous memory value $m_{t-1}$ followed by a second term based on the gradient $g_t$.

Similarly, the velocity $v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2$ uses a one-step memory update term and a squared gradient term. Since $m_t$ and $v_t$ are initialized to zero, to avoid a zero-bias tendency for the first momentum, a bias-corrected version is ${\widehat{m}}_t={\displaystyle \frac{m_t}{1-{\beta }_1^t}}$, and for the second momentum the bias-corrected version is ${\widehat{v}}_t={\displaystyle \frac{v_t}{1-{\beta }_2^t}}$. By considering this, the Adam update formula is:

$$\Delta {\theta }_{t,i}=-{\displaystyle \frac{\eta }{\sqrt{{\widehat{v}}_t}+ϵ}}{\widehat{m}}_t$$

(10)

where a zero division is avoided with $ϵ>0$ in the denominator.

The Adam optimizer is considered the state of the art in deep learning due to its superior performance compared to others, once it computes the decay averages for the first and second moments of the gradients across many neural network layers.

However, a drawback of Adam is the high cost of storing and computing the historical moments.

Several techniques have been proposed for memory reduction in neural networks. For example, Low-Rank Adaptation (LoRA), which is a fine-tuning technique designed to adapt large pre-trained models to specific tasks by freezing a big model and training only small low-rank adapters to reduce gradient’s memory requirements. LoRA is a weight-based approach to save memory [6].

Complementary to LoRA, there are gradient-based approaches, such as Low-Rank Gradient Projection (GaLore), which decomposes the update matrix, to reduce the optimizer state-memory usage by performing Singular Value Decomposition (SVD) on the gradients [7]. It focuses on exploiting the low-rank structure, within the gradient, to achieve memory reduction by projecting the gradient matrix $g_t\in R^{n\times m}$ into a low-dimensional subspace spanned by the columns of a projection matrix $P\in R^{m\times r}$ with $r<<m$. Since gradients are required for parameter updates, the gradient is approximated as $\left(g_{t}P\right)P^{-1}$ by projecting the stored low-dimensional gradient $g_{t}P$ back to the original space [8]. Note that the success of LoRP is the projection to a lower dimension space, and to recover an approximation for the gradients.

In this regard, the wavelet transform is a useful tool which can be applied to reduce the memory required for the moments, based on the principle that energy can be concentrated in fewer coefficients while maintaining the performance of the neural network. A related work [9] presents the Gradient Wavelet Transform (GWT), where the wavelet transform is applied to gradients and significantly reduces the memory requirements of the optimizer at each training step.

The discrete wavelet transform can be defined as:

$$W_{m,n}={\displaystyle \frac{1}{\sqrt{a_0^m}}}\int f\left(x\right)\psi ({\displaystyle \frac{x-n{b}_0{a}_0^m}{a_0^m}})dx$$

(11)

where $a_0>1,b_0>0$ and $m\times n\in {\mathbb{Z}}^2$ [10].

A well-known case is the dyadic wavelet transform where $a_0=2$ and $b_0=1$. In this case, the reconstruction for $f\left(x\right)$ can be expressed as:

$$f\left(x\right)=\sum _{m,n\in \mathbb{Z}}{W}_{m,n}{2}^{-{\displaystyle \frac{m}{2}}}\psi \left(2^{-m}x-n\right)$$

(12)

where $2^{-{\displaystyle \frac{m}{2}}}\psi \left(2^{-m}x-n\right)$ constitutes an orthonormal basis for $L^2\left(\mathbb{R}\right)$ and has good time-frequency localization properties. So, the wavelet coefficients $W_{m,n}$ concentrate the energy in low-frequency components through multiresolution analysis (MRA) [11]. In an MRA, $f\left(x\right)$ has succesive coarser and coarser approximations averaging f over larger intervals, and at every step combining the difference between the approximation with resolution $2^{m-1}$ and the next coarser level, with resolution $2^m$, as a combination of $W_{m,n}$ [10].

The succesive resolution spaces are denoted by $V_m$ where f is approximated as piecewise constant on the interval $[2^{m}n,2^m(n+1)]$. These spaces have the following properties:

1.

$\dots \subset V_2\subset V_1\subset V_0\subset V_{-1}\subset V_{-2}\subset \dots$;

2.

${\cap }_{m\in \mathbb{Z}}{V}_m=\left\{0\right\},\overline{{\cup }_{m\in \mathbb{Z}}{V}_m}=L^2\left(\mathbb{R}\right)$;

3.

$f\in V_j\iff f(2^m·)\in V_0$;

4.

$f\in V_0\to f(·-n)\in V_0$;

5.

There exists $\varphi \in V_0$ so that ${\varphi }_{0,n}\left(x\right)=\varphi (x-n)$, a scaling function, is an orthonormal basis for $V_0$.

The MRA states that, given these five properties, there exists $\psi$ such that $\psi \left(2^{-m}x-n\right)$ constitutes an orthonormal basis. In this way, the approximation $Pro{j}_{V_{m-1}}$ at level $m-1$ is given by:

$$Pro{j}_{V_{m-1}}f=Pro{j}_{V_m}f+\sum _{m\in \mathbb{Z}}<f\left(x\right),\psi (2^{-m}x-n)>\psi \left(2^{-m}x-n\right).$$

(13)

For example, given the Haar basis, it is described by a scaling function $\varphi$:

$$\varphi \left(x\right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}0\le x<1\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(14)

together with a wavelet function $\psi$:

$$\psi \left(x\right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}0\le x<{\displaystyle \frac{1}{2}}\hfill \\ -1,\hfill & \mathrm{if}{\displaystyle \frac{1}{2}}\le x<1\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(15)

Then, the MRA is expressed as:

$$\varphi \left(x\right)=\sum _n{h}_n\phi (2x-n),$$

(16)

$$\psi \left(x\right)=\sum _n{g}_n\varphi (2x-n)$$

(17)

where ${\varphi }_{-1,n}\left(x\right)=\sqrt{2}\varphi (2x-n)$ represents an orthonormal basis for $V_{-1}$ given the sequences $h_n=\sqrt{2}<\varphi ,{\varphi }_{-1,n}>$ and $g_n={(-1)}^n{h}_{-n+1}$.

One of the most powerful results of MRA is the relationship with sub-band filtering, where a digital signal can be decomposed at different resolution levels by iteratively applying filter banks [11]. For the Haar basis, the filter bank is given by a pair of low-pass and high-pass filters. The low-pass filter corresponds to the sequence of coefficients $h_n=\{{\displaystyle \frac{1}{\sqrt{2}}},{\displaystyle \frac{1}{\sqrt{2}}}\}$, whereas the high-pass filter corresponds to the sequence $g_n=\{{\displaystyle \frac{1}{\sqrt{2}}},-{\displaystyle \frac{1}{\sqrt{2}}}\}$. In this sense, the $2\times 2$ Haar matrix is:

$$H_m={\displaystyle \frac{1}{\sqrt{2}}}\left[\begin{array}{cc}1& 1\\ 1& -1\end{array}\right]$$

(18)

which is conformed by two rows $h_n$ and $g_n$, so that it represents a filter bank with perfect reconstruction given that each decomposition level is reversible via the inverse matrix $H_m^{-1}=H_m$.

The discrete wavelet transform for data D is obtained as $H_m·D$, and the reconstruction is given by $H_m^{-1}·H_m·D$. The factor ${\displaystyle \frac{1}{\sqrt{2}}}$ for $H_m$ preserves the norm of the transformed signal and provides perfect reconstruction.

Several perfect reconstruction filters have been studied as generalizations of Haar filters [12]. In particular, length-four filters can be obtained from a set of parametric formulas, such as:

$$\begin{array}{cc}\hfill h_0& ={\displaystyle \frac{1}{2\sqrt{2}}}+{\displaystyle \frac{1}{2}}cos\alpha \hfill \\ \hfill h_1& ={\displaystyle \frac{1}{2\sqrt{2}}}+{\displaystyle \frac{1}{2}}sin\alpha \hfill \\ \hfill h_2& ={\displaystyle \frac{1}{2\sqrt{2}}}-{\displaystyle \frac{1}{2}}cos\alpha \hfill \\ \hfill h_3& ={\displaystyle \frac{1}{2\sqrt{2}}}-{\displaystyle \frac{1}{2}}sin\alpha \hfill \end{array}$$

(19)

which provide $h_n$ coefficients that conform a low-pass filter. The corresponding high-pass filter is given by the set $g_0=h_3,g_1=-h_2,g_2=h_1$ and $g_3=-h_0$. Special cases for Equation (19) are the Haar filter and Dau4 filter, obtained with $\alpha ={\displaystyle \frac{\pi }{4}}$ and $\alpha ={\displaystyle \frac{5\pi }{12}}$, respectively. Other parameterizations have also been studied. For example, length-sixth filters have been studied in [13], length-eight and twelve in [14] and length-sixteen filters have been revised in [15].

Given these parametric filters, it is possible to implement a Parametric Discrete Wavelet Transform (PDWT) that can be applied to the Adam gradients to reduce the memory required for the update step.

For this research, a first goal is to apply the PDWT to the Adam gradients to concentrate the energy in a reduced number of coefficients. Thus, the updates for $m_t$ and $v_t$ are calculated in a reduced wavelet-space, where the gradients do not contain high frequencies. Then, the gradients are restored to the original space to update the network parameters for each training step. Moreover, a second goal is to apply fractional gradients, through the Caputo fractional derivative [16,17,18] to enhance the optimization performance [19,20,21].

Thus, in the present paper, a combination of parametric wavelet transform and fractional gradients are combined to propose FAdamWav, a fractional Adam optimizer where the wavelet transform is applied to reduce the memory of the momentums, and the fractional gradient is used to optimize the free parameters of neural networks.

With this in mind, the rest of the paper is organized as follows: In Section 2, the discrete wavelet transform is revised together with the Caputo fractional derivative and the Adam optimizer. This is the basis for introducing FAdam, an optimizer where a fractional gradient is applied to Adam to optimize the neural network parameters. Subsequently, the parametric wavelet transform is applied to historical momentums of FAdam to obtain FAdamWav, our fractional Adam wavelet-based optimizer.

In Section 3, several experiments are presented to support the main idea that a fractional wavelet optimizer:

• Introduces additional execution time due to the discrete wavelet transform, which has a linear computational complexity;
• Saves memory for the gradients by eliminating high-frequency sub-bands in the wavelet-space;
• Calculates efficiently the momentum $m_t$ and velocity $v_t$ in a low-dimensional wavelet-space;
• Calculates a non-perfect reconstruction for the gradients via the inverse parametric discrete wavelet transform. Then, the recovered approximations of gradients are used to update the neural network parameters;
• Allows to choose a parametric filter for the discrete wavelet transform to provide the best reconstruction for the gradients;
• Enhances the performance compared to its non-fractional wavelet counterparts by considering that fractional gradient optimizer includes the integer case ($\nu =1.0$) as special case;
• Saves gradient’s memory preserving competitive performance compared to the original case without wavelet transformation.

Finally, in Section 4, a discussion takes into account the theoretical concepts and the experimental results to see the advantages and disadvangates of applying our approach. Additionally, future work is commented.

2. Materials and Methods

As previously mentioned, two parts are considered to conform FAdamWav: (i) the parametric discrete wavelet transform that allows to reduce the memory of the Adam gradients for each update step and (ii) the application of the fractional gradient, which uses the Caputo definition of fractional derivative to update the Adam gradients.

2.1. Parametric Discrete Wavelet Transform

The one-dimensional parametric discrete wavelet transform (PDWT) is presented in Algorithm 1, which describes how to implement a perfect reconstruction transformation by applying a pair of parametric filters: a low-pass filter h and a high-pass filter g. The PDWT applies up to K decomposition levels (analysis) to data with $n=2^K$ samples, which correspond to an MRA.

Figure 1 illustrates the PDWT with three decomposition levels. For $k=1$, the sub-band L has low-pass wavelet coefficients (coarse information), whereas the sub-band H stores high-pass coefficients (detailed information). For $k=2$, the second level is only applied to sub-band L, generating $LL$ and $LH$ sub-bands. For $k=3$, the sub-band $LL$ is transformed to obtain bands $LLL$ and $LLH$, and so on, for further levels.

For each level, $k=1,2,\dots ,K$, the number of low-pass wavelet coefficients L is reduced by half. The number of operations for convolution with the h and g filters is twice the length of data at the kth level, which has $2^{K-k+1}$ samples. Therefore, the complexity of the PDWT is ${\sum }_{k=1}^K{2}^k$, yielding an efficient linear complexity of $O\left(n\right)$.

The high-frequency components are in sub-bands with at least one H. The low-frequency sub-band contains only Ls. For two levels, the sub-band $LL$ has low frequencies, and for three levels the $LLL$ sub-band has low frequencies as well. It is worth mentioning that the low-frequency sub-band preserves much of the energy, defined as the sum of the squares of the samples. Additionally, for a single transformation level ($k=1$), erasing the H-band saves $50\%$ of the memory. For $k=2$, erasing H and $LH$ bands saves $75\%$ of the memory. For $k=3$, erasing the H, $LH$, and $LLH$ bands saves $87.5\%$ of the memory, and so on. Although the reconstruction is not perfect, selecting a parametric filter can concentrate more energy in the low-frequency sub-band for the given samples [22], promoting a good approximation to perfect reconstruction.

Algorithm 1 Parametric Discrete Wavelet Transform

Require: Signal array: data, Low-pass filter array of length FILTERSIZE: h, High-pass filter array of length FILTERSIZE: g, Number of wavelet decomposition levels: LEVELS

1: procedure PDWT(data, data.length: n, FILTERSIZE, h, g, LEVELS)

2:  for $\mathtt{level}=0$ to LEVELS do

3:    DWTlevel(data, data.length ≫ level, FILTERSIZE, h, g)

4:  end for

5: end procedure

6: procedure DWTlevel(data, data.length:n, FILTERSIZE, h, g)

7:  if $\mathtt{n}\ge \mathtt{FILTERSIZE}$ then

8:    $i\leftarrow 0$, $j\leftarrow 0$

9:    tmp ← new array of size n                ▹ Temporal array for wavelet coefficients

10:    for $j=0$ to $n-2$ step 2 do

11:     $\mathtt{tmp}\left[i\right]\leftarrow 0$

12:     $\mathtt{tmp}[i+n/2]\leftarrow 0$

13:     for $z=0$ to FILTERSIZE $-1$ do

14:        $\mathtt{tmp}\left[i\right]+=\mathtt{data}\left[\right(j+z)modn]·\mathtt{h}\left[z\right]$               ▹ Low frequency coefficients

15:        $\mathtt{tmp}[i+n/2]+=\mathtt{data}\left[\right(j+z)modn]·\mathtt{g}\left[z\right]$             ▹ High frequency coefficients

16:     end for

17:     $i\leftarrow i+1$

18:    end for

19:    data ← tmp                          ▹ Updated wavelet coefficients

20:  end if

21: end procedure

For reconstruction (synthesis), the same number of levels must be applied as for PDWT. These levels correspond to the Parametric Inverse Discrete Wavelet Transform (PIDWT), which is described in Algorithm 2.

Algorithm 2 Parametric Inverse Discrete Wavelet Transform

Require: Signal array: data, Low-pass filter array of length FILTERSIZE: Ih, High-pass filter array of length FILTERSIZE: Ig, Number of wavelet decomposition levels: LEVELS

1: Ih, Ig ← InverseFilters(FILTERSIZE, h, g)

2: procedure PIDWT(data, data.length: n, FILTERSIZE, Ih, Ig, LEVELS)

3:  for $level\leftarrow \mathtt{LEVELS}$ downto 0 do

4:     INVTWDLEVEL(data, data.length ≫ level, FILTERSIZE, h, g)

5:  end for

6: end procedure

7: procedure INVTWDLEVEL(data, data.length, FILTERSIZE, Ih, Ig)

8:  if $n\ge filtersize$ then

9:     $ndiv2\leftarrow n/2$

10:     $tmp\leftarrow \mathrm{new}\mathrm{array}\mathrm{of}\mathrm{size}n$

11:     $j\leftarrow FILTERSIZE-2$

12:     for $i\leftarrow 0$ to $ndiv2-1$ do

13:     $tmp[jmodn]\leftarrow 0$

14:     for $w\leftarrow 0,q\leftarrow 0$ to $filtersize/2-1$ do

15:       $ii\leftarrow (i+w/2)modndiv2$

16:       $tmp[jmodn]+=data\left[ii\right]·Ih\left[w\right]$

17:       $tmp[jmodn]+=data[ii+ndiv2]·Ih[w+1]$

18:       $w\leftarrow w+2$

19:     end for

20:     $j\leftarrow j+1$

21:     $tmp[jmodn]\leftarrow 0$

22:     for $w\leftarrow 0$ to $FILTERSIZE-1$ step 2 do

23:       $tmp[jmodn]+=data\left[\right(i+w/2)modndiv2]·Ig\left[w\right]$

24:       $tmp[jmodn]+=data\left[\right(i+w/2)modndiv2+ndiv2]·Ig[w+1]$

25:     end for

26:     $j\leftarrow j+1$

27:     end for

28:     $data\leftarrow tmp$

29:  end if

30: end procedure

For each transformation level the number of wavelet coefficients is duplicated up to the original via the PIDWT using two reconstruction filters $Ih$ and $Ig$ obtained from h and g, according to Algorithm 3.

Algorithm 3 Reconstruction filters

1: function InverseFilters($FILTERSIZE,h,g$)

2:  for $i\leftarrow 0$ to $FILTERSIZE-1$ do

3:     $base\leftarrow \left\{\begin{array}{ll}h& \mathrm{if} i \mathrm{is} \mathrm{even}\\ g& \mathrm{if} i \mathrm{is} \mathrm{odd}\end{array}\right.$

4:     $Ih\left[i\right]\leftarrow base[2·\lfloor (FILTERSIZE-1-i)/2\rfloor ]$

5:     $Ig\left[i\right]\leftarrow base[2·\lfloor (FILTERSIZE-1-i)/2\rfloor +1]$

6:  end for

7:  return $Ih\left[i\right]$, $Ig\left[i\right]$

8: end function

The shortest filter has a length of two and corresponds to the Haar filter. Parametric length-four filters are shown in Equation (19).

Length-sixth filters are described in Equation (20), depending on two parameters $\alpha$ and $\beta$ as follows:

$$\begin{array}{cc}\hfill p& =0.5\ast \sqrt{1+sin(\alpha +{\displaystyle \frac{\pi }{4}})}\hfill \\ \hfill h\left[0\right]& =\sqrt{2}·\left({\displaystyle \frac{1}{8}}+{\displaystyle \frac{1}{4\sqrt{2}}}cos\left(\alpha \right)+{\displaystyle \frac{p}{2}}cos\left(\beta \right)\right)\hfill \\ \hfill h\left[1\right]& =\sqrt{2}·\left({\displaystyle \frac{1}{8}}+{\displaystyle \frac{1}{4\sqrt{2}}}sin\left(\alpha \right)+{\displaystyle \frac{p}{2}}sin\left(\beta \right)\right)\hfill \\ \hfill h\left[2\right]& =\sqrt{2}·\left({\displaystyle \frac{1}{4}}-{\displaystyle \frac{1}{2\sqrt{2}}}cos\left(\alpha \right)\right)\hfill \\ \hfill h\left[3\right]& =\sqrt{2}·\left({\displaystyle \frac{1}{4}}-{\displaystyle \frac{1}{2\sqrt{2}}}sin\left(\alpha \right)\right)\hfill \\ \hfill h\left[4\right]& =\sqrt{2}·\left({\displaystyle \frac{1}{8}}+{\displaystyle \frac{1}{4\sqrt{2}}}cos\left(\alpha \right)-{\displaystyle \frac{p}{2}}cos\left(\beta \right)\right)\hfill \\ \hfill h\left[5\right]& =\sqrt{2}·\left({\displaystyle \frac{1}{8}}+{\displaystyle \frac{1}{4\sqrt{2}}}sin\left(\alpha \right)-{\displaystyle \frac{p}{2}}sin\left(\beta \right)\right).\hfill \end{array}$$

(20)

It is worth mentioning that special cases for Equation (19) are Haar and Dau4, as is shown in

Table 1. Special cases for length-four parametric filters.

Filter	$\mathit{\alpha }$
Haar	0.7853981633974483
Dau4	1.3089969389957472

, with the corresponding parameter $\alpha$ [23].

Additionally, there are special cases of length-sixth parametric filters of Equation (20) for Haar, Dau4, Dau6, and Coiflets, which are listed in

Table 2. Special cases for length-sixth parametric filters.

Filter	$\mathit{\alpha }$	$\mathit{\beta }$
Haar	0.7853981633974483	0.7853981633974483
Dau4	1.3089969389957472	1.0471975511965976
Dau6	1.7850806990394015	1.0742468359786252
Coiflets	5.7504673989251085	3.6052402624389956

with the corresponding $\alpha$ and $\beta$ parameters [23].

By selecting one of these parameterizations for the low-pass h and high-pass g filters, the parametric discrete wavelet transform can be applied to the gradients of the Adam optimizer.

2.2. Fractional Derivatives and the Backpropagation Update Formula for MLP

There is no single definition of fractional derivatives. Three notable examples are the Grünwald–Letnikov, the Riemann–Liouville, and the Caputo fractional derivatives [16,19,24] which are defined and described briefly below.

Definition 1.

The Grünwald–Letnikov derivative with order $\nu >0$ of a given function $f\left(x\right),x\in (a,b)$ is defined as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {}_a{}^{GL}D_x^{\nu }f\left(x\right)=\underset{\begin{array}{c}h\to 0\\ Nh=x-a\end{array}}{lim}{h}^{-\nu }\sum _{j=0}^N{(-1)}^j\left({\displaystyle \genfrac{}{}{0pt}{}{\nu }{j}}\right)f(x-jh).\end{array}\end{array}$$

(21)

Definition 2.

The Riemann–Liouville derivative with order $\nu >0$ and $n=[\nu +1]$ of a given function $f\left(x\right),x\in (a,b)$ is defined as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {}_a{}^{RL}D_x^{\nu }f\left(x\right)={\displaystyle \frac{1}{\Gamma (n-\nu )}}{\displaystyle \frac{d^n}{d{x}^n}}{\int }_a^x{(x-y)}^{n-\nu -1}f\left(y\right)dy.\end{array}\end{array}$$

(22)

Definition 3.

The Caputo fractional derivative, for $a,x\in \mathbb{R}$, $\nu >0$ and $n=[\nu +1]$, is defined as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {}_a{}^{C}D_x^{\nu }f\left(x\right)={\displaystyle \frac{1}{\Gamma (n-\nu )}}{\int }_a^x{(x-y)}^{n-\nu -1}{f}^{\left(n\right)}\left(y\right)dy.\end{array}\end{array}$$

(23)

Although these fractional derivatives are certainly related, they are not equivalent [25]. For example, the Riemann–Liouville definition is related to the Caputo definition as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {}^{RL}{}_{a}D_x^{\nu }f\left(x\right){=}_a^C{D}_x^{\nu }f\left(x\right)+\sum _{k=0}^{n-1}{\displaystyle \frac{f^{\left(k\right)}\left(a\right){(x-a)}^{k-\nu }}{\Gamma (k+1-\nu )}},\end{array}\end{array}$$

(24)

and, for a suitably smooth function $f\left(x\right)$, i.e., $f\in C^m[a,b]$, then

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {}^{RL}{}_{a}D_x^{\nu }f\left(x\right)={}^{GL}{}_{a}D_x^{\nu }f\left(x\right).\end{array}\end{array}$$

(25)

From the definition of the Riemann–Liouville derivative, it follows that if $f\left(x\right)=k$ is a non-zero constant, then:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {}^{RL}{}_{0}D_x^{\nu }f\left(x\right)={\displaystyle \frac{k}{\Gamma (1-\nu )}}{x}^{-\nu },\end{array}\end{array}$$

(26)

which is not zero and is different from the integer case.

In this paper and other works, the Caputo definition is preferable to the Grünwald–Letnikov and Riemman–Liouville definitions because the Caputo fractional derivative of Equation (23) is zero for $f\left(x\right)=k$, with $k\in \mathbb{R}$. This matches the integer derivative version [18].

Based on the previous discussion, we use the Caputo derivative definition to obtain a fractional expression for the gradients of the Adam algorithm. As we pointed out, the Caputo derivative of a constant is zero and for $f\left(x\right)=x^p$ [18,26]:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {}_0{}^{C}D_x^{\nu }{x}^p={\displaystyle \frac{\Gamma (p+1)x^{p-v}}{\Gamma (p-v+1)}}.\end{array}\end{array}$$

(27)

In the case of $p=1$, it yields to:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {}_0{}^{C}D_x^{\nu }x={\displaystyle \frac{x^{1-v}}{\Gamma (2-v)}}.\end{array}\end{array}$$

(28)

Given this fractional derivative and the backpropagation formula which is used to update the parameters of the Multilayer Perceptron (MLP) [1]:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \Delta w_{kj}^l=-\eta {\displaystyle \frac{\partial E_i}{\partial w_{kj}^l}}=-\eta D_{w_{kj}^l}{E}_i\end{array}\end{array}$$

(29)

our fractional version to update the gradients for FAdamWav is:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill D_{w_{kj}^l}^{\nu }{E}_i={\displaystyle \frac{\partial E_i}{\partial w_{kj}^l}}·D_{w_{kj}^l}^{\nu }{w}_{kj}^l={\delta }_k^l·a_j^{l-1}·{\displaystyle \frac{{\left(w_{kj}^l\right)}^{1-\nu }}{\Gamma (2-\nu )}}.\end{array}\end{array}$$

(30)

See Appendix A for details. Now, it is worth making a comment in this regard. In the context of the calculus of paradifferential operators in Sobolev spaces, and in this case for the Fourier multiplier operator $D^{\nu }$, defined by:

$$\widehat{\left(D^{\nu }f\right)}\left(\xi \right)={\left|\xi \right|}^{\nu }\widehat{f}\left(\xi \right),$$

(31)

the fractional chain rule [27,28] should be:

$$D^{\nu }F\left(u\right)=F^{\prime }\left(u\right)D^{\nu }u+E,$$

(32)

where E is an error as per the result of Christ and Weinstein [29]. Thus, by assuming $E=0$, it is an approximation, useful for our purposes. Suppose that $F\in C^1\left(\mathbb{C}\right)$, $1<p,q<\infty$, $\nu \in (0,1)$, $r^{-1}=p^{-1}+q^{-1}$. If $u\in L^{\infty }\left(\mathbb{R}\right)$, $D^{\nu }u\in L^q$, and $F^{\prime }\left(u\right)\in L^p$, then $(D^{\nu }\left(F\left(u\right)\right)\in L^r$, and $\left|\right|D^{\nu }{F\left(u\right)\left|\right|}_r\le C\left|\right|F^{\prime }{\left(u\right)\left|\right|}_p\left|\right|D^{\nu }\left(u\right){\left|\right|}_q$.

It is important to note that Equation (30) is the fractional version of Equation (A5), only different by the factor ${\displaystyle \frac{{\left(w_{kj}^l\right)}^{1-\nu }}{\Gamma (2-\nu )}}$ which is obtained when Equation (28) is applied to $w_{kj}^l$. Indeed, if $\nu =1$, Equation (30) becomes the classical integer case. Then, Equation (30) represents a gradient-descent generalization with fractional order $\nu$.

In practice, it is necessary to avoid two conditions:

• When the synaptic weights are zero, that yields to the indetermination of ${\displaystyle \frac{{(w_{kj}^l)}^{1-\nu }}{\Gamma (2-\nu )}}$ for $1-\nu <0$.
• When $1-\nu$ is rational, we let $1-\nu ={\displaystyle \frac{r}{s}}$ and s is even (for example $r=1$ and $s=2$). Then, if $w_{kj}^l<0$, complex values are generated.

These situations have been previously studied in [20] and a solution consists of replacing $w_{kj}^l$ by $|w_{kj}^l|+ϵ$, for $ϵ>0$. In this way, the fractional gradient factor $f_w^{\nu }$ is defined as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill f_w^{\nu }:={\displaystyle \frac{\left(\right|w_{kj}^l{|+ϵ)}^{1-\nu }}{\Gamma (2-\nu )}}\end{array}\end{array}$$

(33)

and the limit exists, and is equal to 1, for $f_w^{\nu }$ as $\nu \to 1$.

With this consideration, Equation (30) becomes:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill D_{w_{kj}^l}^{\nu }{E}_i={\delta }_k^l·a_j^{l-1}·f_w^{\nu }={\delta }_k^l·a_j^{l-1}·{\displaystyle \frac{\left(\right|w_{kj}^l{|+ϵ)}^{1-\nu }}{\Gamma (2-\nu )}}\end{array}\end{array}$$

(34)

which generalizes the integer gradient-descent update rule, and it allows to compute the fractional gradient for the Adam optimizer by including the factor $f_w^{\nu }$.

The following Algorithm 4 presents the FAdamWav steps. In this case, the gradient $g_t$ is a bidimensional $m\times n$ matrix. In Step 1, the fractional gradient is obtained by applying Equation (34). To warrant a two-power for the n-dimension, zero padding is applied to obtain $n=2^K$ samples. Then, the PDWT is applied to m-tensors and therefore the computational complexity to calculate the PDWT is $O(m\times n)$.

In this paper, it is meaningless to assume that a 2D wavelet transform on $g_t$ could be better to exploit some kind of “curves” inside $g_t$. Instead, we just consider m transformations on n-dimensional tensors. However, we mention in brief a 2D approach, where a PWDT could be applied in two separated dimensions through tensor products of parametric filters or from non-tensor product of wavelet filters [30]. Indeed, it is possible to consider curvelets [31] and shearlets [32]. Curvelets are efficient transforms designed to work on images with edges along smooth curves, and they extend wavelets by incorporating directional selectivity and anisotropic scaling. The shearlet transform was designed to capture anisotropic features (like edges) in multidimensional data. We remark that these approaches are not considered in this paper, but it could be explored in future work.

Continuing with the FAdamWav algorithm (Algorithm 4), Step 5 shows $L_t$ which represents the low-frequency band (only “L’s”) along T iterations and the $H_t$ sub-bands with high-frequency components, for k transformation levels on $g_t$. After applying the PDWT, the $H_t$ sub-bands are erased to save memory. Then, the $m_t$ and $v_t$ matrices are initialized to zero, and they are updated with $L_t$ in Steps 10 and 11, by considering Equations (8) and (9), respectively. $L_t$ has dimension of $m\times 2^{K-k+1}$ and $m_t$ and $v_t$ also save space since they have the same dimension. $L_t$ reduces the low-frequency coefficients by half for each level. For example, when $k=1$, the memory of the fractional gradients is reduced by $50\%$, $g_t$ has a dimension $m\times {\displaystyle \frac{m}{2}}$, and $H_t$ is set to zero in Step 6. So, in Step 12, $L_t$ is calculated using Equation (10) according to Adam’s update formula. One difference from related work [9] is that our algorithm fully erases $H_t$ and is not scaled by the formula ${\displaystyle \frac{H_t}{\sqrt{v_t}+ϵ}}$ which scales the high-frequency sub-bands with the $v_t$ matrix to deal with noisy gradients. In our research, we assume the existence of an optimal parametric filter to compress the gradients, allowing for non-perfect reconstruction of $g_t$ to update $W_t$ parameters, as shown in Step 15. The main idea is that it is possible to achieve a competitive accuracy relative to the original $g_t$ saving memory.

Algorithm 4 FAdamWav: Fractional Adam with Wavelet Transform

Require: Weight matrix W, learning rate $\eta$, batch size m. Adam decay rates ${\beta }_1,{\beta }_2$. Iteration T. $ϵ>0$. Scale factor $\alpha$. Lowpass wavelet filter h, highpass wavelet filter g, Levels k.

1: Initialize $t\leftarrow 0$

2: repeat

3:  $g_t\leftarrow g_t·f_w^{\nu }$                       ▹ Fractional gradient. See Equation (34)

4:  $g_t\leftarrow pad\_to\_pow2(g\_t)$                ▹ Save original length. Length($g_t$) $=2^K$

5:  $[L_t,H_t]\leftarrow PDWT(g_t,length\left(g_t\right),filtersize,h,g,k)$             ▹ Gradient PDWT- Analysis

6:  $H_t\leftarrow 0$                          ▹ No longer used, memory reduction

7:  if $t=0$ then

8:    Initialize $m_{-1},v_{-1}\leftarrow 0$

9:  end if

10:  $m_t\leftarrow {\beta }_1{m}_{t-1}+(1-{\beta }_1)L_t$

11:  $v_t\leftarrow {\beta }_2{v}_{t-1}+(1-{\beta }_2)L_t^2$

12:  $[L_t,H_t]\leftarrow \left[{\displaystyle \frac{m_t}{\sqrt{v_t}+ϵ}},{\displaystyle \frac{H_t}{\sqrt{v_t}+ϵ}}\right]$         ▹$H_t$ was set to zero. Update rule, see Equation (10)

13:  $g_t\leftarrow PIDWT([L_t,H_t],length(L_t,H_t),filtersize,Ih,Ig,k)$ ▹ PIDWT - Synthesis. Back to the original length

14:  ${\eta }_t\leftarrow \eta ·{\displaystyle \frac{\sqrt{1-{\beta }_2^t}}{1-{\beta }_1^t}}$                              ▹ Bias correction

15:  $W_t\leftarrow W_{t-1}-{\eta }_t·g_t$         ▹ Update weights. Original dimensions for fractional gradients.

16:  $t\leftarrow t+1$

17: until $t=T$

18: return $W_t$

Based on the previous concepts and Algorithm 4, the original PyTorch code for the Adam class [33] can be modified to obtain our fractional FAdamWav class, which is described in Listing 1. Specifically, by inserting Lines 8 to 20 as follows:

Listing 1. FAdamWav Class: PyTorch Source Code

1 class FAdamWav(Optimizer):  2  def step(self, closure: Callable = None):  3   …  4   for group in self.param_groups:  5    for p in group["params"]:  6     …  7     grad = p.grad  8     grad *= torch.pow(abs(grad)+ epsilon, 1-nu )  9     grad /= torch.exp(torch.lgamma(torch.tensor( 2.0-nu )  )) 10     gr, original_lengths  = pad_batch_to_pow2(grad) 11     pwt = ParametricWaveletTransform () 12     gr = pwt.PDWT(gr, len(gr)) 13     gr = gr[…, :gr.shape[−1] // 2] 14     grad = gr 15     … 16     gr = norm_grad 17     gr = duplicate_with_zeros(gr) 18     gr = pwt.PIDWT(gr, len(gr)) 19     gr = restore_original_lengths(gr, original_lengths) 20     grad = gr.squeeze(0) 21     norm_grad = grad

In this case, the fractional factor depending of the $\nu$-order is applied to $grad$ (see Lines 8 and 9), and consequently it becomes a fractional gradient. (The source code for FAdamWav, PDWT, and PIDWT is available at http://ia.azc.uam.mx/sourcecode) (accessed on 23 February 2026).

In the same code, Lines 10 to 20 apply the parametric discrete wavelet transform. Line 12 applies the PDWT. Note that Line 13 considers low-pass wavelet coefficients exclusively, under the principle that these coefficients should concentrate much of the gradient energy (high-pass wavelet coefficients are neglected) which reduces the memory required for the gradients as previously explained. In Line 17, after using the gradients to update the synaptic weights, the high-pass wavelet coefficients $H_t$ are set to zero, and the PIDWT is applied in Line 18. Of course, this produces a non-perfect reconstruction, but it is sufficient to update the gradients of Adam since $m_t$ and $v_t$ are updated efficiently in the transformed space. The accuracy of the neural network should be lower than using the original gradient $g_t$. However, the main idea is to maintain competitiveness while saving memory. This idea is supported by experimental results.

3. Results

This section describes four experiments that support the conclusions.

• Experiment 1. (FAdam). It explores the accuracy of a neural network with the following architecture:
  • Linear(784, 64)
  • ReLU()
  • Linear(64, 10)
  and fractional gradients $\nu \in (0,2.0)$ on the MNIST dataset [34], without applying the parametric discrete wavelet transform. The neural network architecture is enough to match the input size of MNIST images ($28\times 28$ pixels) and the output with 10 classes. The MNIST dataset has 60,000 images, divided into a training set of 50,000 and a test set of 10,000.
• Experiment 2. (FAdamWav: Length-four filters). Given the same neural network and dataset as in Experiment 1, the second experiment applies the parametric wavelet transform to reduce the FAdam gradient memory by half by considering only the low-pass coefficients and a single transformation level. Several length-four parametric filters are tested varying the parameter $\alpha =j·{\displaystyle \frac{\pi }{12}}$ for $j=\{1,2,3,4,5,6\}$. In this case, the best $\nu$-value from Experiment 1 is chosen (see below $\nu =0.2$).
• Experiment 3. (FAdamWav: Length-sixth filters). This experiment is similar to Experiment 2, but it considers length-sixth filters of Equation (20) depending on $\alpha$ and $\beta$. Each parameter independently takes on values equal to $j·{\displaystyle \frac{\pi }{12}}$, where $j=\{1,2,3,4,5,6\}$.
• Experiment 4. (FAdam, FAdamWav_k1, FAdamWav_k2, FAdamWav_k3). This experiment measures and compares the amount of GPU memory required for Adam and FAdamWav applying $k=1,2,3$ wavelet transformation levels. The execution time is also compared.

3.1. Experiment 1

Table 3. Neural Network accuracy for different $\nu$-values with the FAdam optimizer.

$\mathit{v}=0.2$	$\mathit{v}=0.4$	$\mathit{v}=0.6$	$\mathit{v}=0.8$	$\mathit{v}=1.0$	$\mathit{v}=1.2$	$\mathit{v}=1.4$	$\mathit{v}=1.6$	$\mathit{v}=1.8$
96.25	96.46	96.53	96.04	96.44	96.34	96.89	95.84	96.03
97.12	97.11	97.37	97.54	97.09	97.24	96.51	97.04	96.93
97.49	97.28	97.30	97.54	97.51	97.47	97.15	97.41	96.99
97.60	97.63	97.63	97.67	97.66	97.55	97.43	97.33	97.10
97.63	97.77	97.75	97.88	97.37	98.18	97.98	97.86	97.23
97.33	97.69	97.63	98.08	97.63	97.88	97.69	97.63	97.29
97.68	97.64	97.78	97.48	97.66	97.95	97.57	97.61	97.39
97.76	97.69	97.80	97.59	97.59	97.72	97.84	97.60	97.39
97.88	97.26	97.87	98.01	98.05	97.94	97.83	97.88	97.71
97.84	97.36	98.00	98.10	97.88	97.84	98.09	98.05	97.45
97.65	97.69	97.63	97.27	97.69	97.98	97.89	97.94	97.23
97.86	98.07	98.00	97.90	97.48	98.04	97.91	97.86	97.59
98.00	97.75	97.88	97.92	97.84	97.98	97.65	97.74	97.76
98.07	97.85	97.96	98.02	97.73	97.96	97.67	97.81	97.63
97.68	97.55	97.91	98.18	97.99	97.72	97.89	97.63	97.41
97.99	97.91	97.95	97.67	97.73	97.88	97.83	97.63	97.59
97.91	98.01	97.82	97.55	97.88	97.94	97.86	97.85	97.84
97.93	98.12	97.90	98.04	97.86	97.55	97.76	97.98	97.73
98.36	98.12	97.94	98.23	98.04	98.05	98.06	97.83	97.75
97.80	97.94	97.74	97.90	98.00	97.89	98.09	97.94	97.58

shows the accuracy of the neural network trained using FAdam and different fractional gradients $\nu \in (0,2.0)$, with step $0.2$ along 20 epochs. The maximum accuracy of $98.36\%$ is reached with $\nu =0.2$, which is highlighted in bold.

Figure 2 shows boxplots of the accuracies of Experiment 1 (with a step size of 0.1 for $\nu$), showing that the best performance is achieved with $\nu =0.2$. Additionally, the plot shows that the integer case $\nu =1.0$ has lower performance than several fractional cases, including the case $\nu =0.2$.

To complete Experiment 1, Figure 3 compares the performance of the integer case $\nu =1.0$ against the case $\nu =0.2$. It is notorious that the fractional gradient reaches a higher accuraccy than the integer gradient.

3.2. Experiment 2

The second experiment applies a single level of the parametric wavelet transform to reduce the FAdam gradient memory by half and by considering only the low-pass wavelet coefficients. Then, the gradients are updated, the high-pass coefficients are set to zero, and the parametric inverse wavelet transform is calculated to recover the original size of the Adam gradients.

Several length-four parametric filters were tested by varying the parameter $\alpha =k·{\displaystyle \frac{\pi }{12}}$ for $k=\{1,2,3,\dots ,12\}$. The $\nu$-value was fixed to $\nu =0.2$. The corresponding accuracies are shown in columns of

Table 4. Neural Network accuracy for $\nu =0.2$ with FAdamWav optimizer: Several $\alpha$ values.

$\frac{\mathit{\pi }}{12}$	$\frac{\mathit{\pi }}{6}$	$\frac{\mathit{\pi }}{4}$	$\frac{\mathit{\pi }}{3}$	$\frac{5\mathit{\pi }}{12}$	$\frac{\mathit{\pi }}{2}$	$\frac{7\mathit{\pi }}{12}$	$\frac{2\mathit{\pi }}{3}$	$\frac{3\mathit{\pi }}{4}$	$\frac{5\mathit{\pi }}{6}$	$\frac{11\mathit{\pi }}{12}$	$\mathit{\pi }$
94.90	95.33	95.78	95.61	94.95	95.33	95.67	95.14	95.41	95.49	95.36	95.64
96.47	96.62	96.62	96.65	96.80	96.63	96.80	96.85	96.75	96.57	96.57	96.89
96.91	97.28	97.33	97.06	97.09	97.15	97.36	97.13	97.09	97.22	97.20	97.35
97.47	97.55	97.60	97.25	97.10	97.31	97.55	97.24	97.23	97.59	97.42	97.22
97.14	97.59	97.49	97.78	97.10	97.69	97.23	97.47	97.38	97.23	97.38	97.75
97.50	97.74	97.42	97.86	97.82	97.77	97.85	97.90	97.37	97.68	97.21	97.58
97.44	97.12	97.45	97.88	97.62	97.96	97.93	97.45	97.63	97.63	97.85	97.75
97.76	97.48	97.48	97.68	97.08	97.53	97.81	97.61	97.71	97.47	97.42	97.79
97.87	97.88	97.60	97.59	97.75	97.75	97.21	97.74	97.81	97.49	97.76	97.26
97.80	97.84	97.73	97.81	97.82	97.79	97.90	97.52	97.56	97.85	97.74	97.97
97.72	97.80	97.88	97.99	97.48	97.73	97.57	97.40	97.49	97.81	97.67	97.69
97.81	97.64	97.74	97.54	97.48	97.75	97.75	97.91	97.72	97.87	97.78	97.61
97.94	97.84	97.66	98.13	97.85	97.84	97.84	97.92	97.97	97.96	97.63	97.78
97.81	97.95	97.61	97.68	97.87	97.60	97.50	97.86	97.74	97.90	97.72	97.69
98.01	97.78	97.06	97.96	97.54	98.06	97.85	97.88	97.83	97.69	97.82	97.86
97.88	97.72	98.00	97.92	97.27	98.07	97.51	97.97	97.77	97.80	97.72	97.83
97.79	97.66	98.15	98.03	97.82	98.05	98.03	97.98	97.92	97.85	97.99	97.94
97.89	97.85	97.77	97.95	98.00	97.67	97.95	97.73	97.90	97.92	97.51	97.77
97.84	98.16	97.97	97.89	97.86	98.03	97.96	97.78	97.70	98.04	98.07	98.04
97.85	97.99	97.88	97.24	97.98	97.84	97.95	97.91	97.86	98.01	97.90	98.01

, where the highest accuracy $98.16\%$ was achieved with $\alpha ={\displaystyle \frac{\pi }{6}}$ and is highlighted in bold.

Figure 4 shows the boxplots of Table 4, and effectively it illustrates that the best performance occurs at $\alpha ={\displaystyle \frac{\pi }{6}}$.

Figure 5 compares the accuracies of Fractional Adam (FAdam) with $\nu =0.2$ and Fractional Adam with parametric wavelet transform (FAdamWav) with $\nu =0.2$ and a parametric filter with $\alpha ={\displaystyle \frac{\pi }{6}}$. Although FAdam has a slightly better performance than FAdamWav, performance is still competitive but the memory required by the gradients is reduced by half.

3.3. Experiment 3

Experiment 3 considers FAdamWav with fractional order $\nu =0.2$ and the parametric length-sixth filters given in Equation (20) depending on $\alpha$ and $\beta$. In the experiment, ${\alpha }_i=i·{\displaystyle \frac{\pi }{12}}$ and ${\beta }_j=j·{\displaystyle \frac{\pi }{12}}$ with $i,j\in \{1,2,3,4,5,6\}$.

The experiments were developed in PyTorch 2.3.1, running on a GPU NVIDIA GeForce RTX 3070. The Adam optimizer used a learning rate $lr=0.01$ and the test accuracy was used as the metric for comparison.

Figure 6 shows the accuracy of each parametric filter (${\alpha }_i,{\beta }_j$). The best result is for ${\alpha }_i={\displaystyle \frac{3\pi }{12}}$ and ${\beta }_j={\displaystyle \frac{5\pi }{12}}$. The experiment reveals that the filter modifies the training performance of the neural network.

3.4. Experiment 4

In Experiment 4, the amount of GPU memory used during executions was measured. These executions developed a comparison with and without applying the wavelet transform: “FAdam” refers to the execution without wavelets whereas “FadamWav_k1”, “FAdamWav_k2”, “FAdamWav_k3” correspond to $k=1,2$, and 3 wavelet transformation levels, respectively.

Note that theoretically, the memory savings are $50\%$ for FAdamWav_k1, $75\%$ for FAdamWav_k2, and $87.5\%$ for FAdamWav_k3. However, as shown in

Table 5. Gradient memory usage: FAdam vs. FAdamWav for $k=1,2$, and 3 wavelet transformation levels.

Memory Saving	FAdam	FAdamWav_k1	FAdamWav_k2	FAdamWav_k3
Theoretical (percentage)	0 MB	50% MB	75% MB	87.5%
GPU memory usage (practical measure MB)	28.8 MB	23.95 MB	18.82 MB	18.82 MB
Practical (percentage)	0 MB	16.8% MB	34.6% MB	34.6%

, the reduction in GPU memory using wavelets does not correspond to these percentages when compared to FAdam as the reference.

Nevertheless, there is a reduction in GPU memory usage: from $28.8$ MB with FAdam to $23.95$ MB with FAdamWav_k1, which represents a $16.8\%$ reduction. A second transformation level reduces the GPU memory usage from $28.8$ MB to $18.82$ MB (a $34.6\%$ reduction). With three transformation levels, the memory usage remains at $18.82$ MB. Therefore, there is no further savings from two to three transformation levels. This is because GPU memory is assigned in blocks, so when memory usage does not change considerably, the reported memory usage remains the same.

Figure 7 shows the accuracy for FAdam (without applying wavelets), FAdamWav_k1, FAdamWav_k2, and FAdamWav_k3, during 20 epochs. In Figure 7, it is possible to appreciate that:

• FAdamWav_Level1 has a similar performance compared to FAdam, even though FAdamWav_Level1 saves $16.8\%$ of GPU memory.
• FAdamWav_Level2 has lower accuracy than FAdam and FAdamWav_k1, but it is still competitive despite the $36.4\%$ reduction in GPU memory.
• FAdamWav_Level3 has lower performance than FAdamWav_Level2, and no more GPU memory reduction.

The accuracies for FAdam, FAdamWav_k1, FAdamWav_k2, and FAdamWav_k3 are shown in

Table 6. Accuracy: Fadam vs. FadamWav for $k=1,2$, and 3 wavelet transformation levels.

Adam	FAdamWav_k1	FAdamWav_k2	FAdamWav_k3
95.98	95.11	94.59	93.31
96.83	96.56	95.65	95.04
96.62	97.20	96.19	96.15
97.68	97.64	96.80	96.32
97.10	97.07	96.52	96.21
97.46	97.59	97.15	96.73
97.80	97.56	96.88	96.90
97.69	97.81	97.23	96.92
97.64	97.73	97.26	97.03
97.28	97.84	97.63	97.36
97.70	97.58	97.19	97.12
97.49	97.73	97.60	97.14
98.00	97.91	97.60	97.30
97.78	97.99	97.65	97.45
97.98	97.98	97.59	97.49
97.88	97.93	97.59	97.47
97.70	97.63	97.84	97.49
97.81	98.08	97.70	97.49
98.06	97.94	97.45	97.66
97.90	97.82	97.61	97.49

.

The elapsed time for the 20 epochs whose data are reported in Table 6 are shown in

Table 7. Elapsed time: FAdam vs. FAdamWav for $k=1,2$, and 3 wavelet transformation levels.

FAdam	FAdamWav_Level1	FAdamWav_Level2	FAdamWav_Level3
131 s	181 s	201 s	225 s

, where it is possible to see that:

• The application of a single wavelet transformation level (FAdamWav_k1) increases the execution time for 50 s with respect to FAdam.
• The application of a second wavelet transformation level (FAdamWav_k2) increses the execution time for 70 s with respect to FAdam.
• The application of a third wavelet transformation level (FAdamWav_k3) increses the execution time to 94 s with respect to FAdam.

These experiments suggest that applying one or two transformation levels reduces GPU memory while maintaining competitive performance. However, using more than two transformation levels is not beneficial because it increases the execution time without providing additional memory reduction.

4. Discussion

The optimizer is a crucial component of neural network training. Due to the rapid growth of Artificial Intelligence, several gradient-based optimizers have been proposed. The Adam optimizer is considered the state of the art in deep learning because it outperforms other optimizers by considering historical adaptations of the moments. However, a drawback is the cost of storing and computing these gradients.

In this regard, we present FAdamWav, a variant of Adam which combines wavelet-based gradient compression with fractional gradient optimization.

The main idea behind FAdamWav is to reduce the memory required for the gradients by compressing them with a discrete wavelet transform while maintaining competitive accuracy. It also aims to improve performance by leveraging the fact that the fractional gradient optimizer outperforms the integer case. The integer case is a special case of the fractional optimizer when $\nu =1.0$.

FAdamWav is characterized by the following:

1.

Wavelet gradient compression. A wavelet-based gradient compression maps the gradient matrix to a reduced space via the parametric discrete wavelet transform with k transformation levels.

2.

Efficient calculation of moments and velocities. Calculating the moments $m_t$ and velocities $v_t$ is efficient because high-frequency bands are removed in the wavelet low-dimensional space. Therefore, the gradients do not contain high frequencies.

3.

Fractional gradient optimization. The fractional $\nu$-order of the optimizer is variable. Selecting the appropriate value of $\nu$ provides good optimization. Our experiments show that the integer case is not optimal, and that the fractional version outperforms it. For instance, in Experiment 1, the best fractional order was $\nu =0.2$.

4.

Parametric filter selection. FAdamWav allows to select optimal parameter values of the filters for the discrete wavelet transform. The goal is to achieve maximum compression so that, even when the high-frequency coefficients are set to zero, the reconstructed gradients are sufficiently similar to the original.

Although the experiments were developed by varying the parameters step by step with discrete values in the range $(0,2\pi )$, it is possible to adjust the $\nu$-order during execution.

5.

Neural network parameter optimization. The inverse parametric discrete wavelet transform restores the gradients to their original dimensions. Once restored, the gradients are used to update the network parameters for each training step.

6.

Wavelet transformation levels. FAdamWav has the possibility of applying k transformation levels, and it is possible to choose a value of k that provides an acceptable trade-off between longer execution time and lower memory usage.

7.

One-dimensional discrete wavelet transform. The discrete wavelet transform is applied in one dimension. However, it is possible to deal with the gradient matrix $g_t$ using a two dimensional wavelet transform, including curvelets or shearlets.

As previously mentioned, introducing the wavelet transform increases execution time, but this is an acceptable trade-off given the memory reduction.

Although the memory savings do not reach the theoretical limits of $50\%$ or $75\%$ for one and two transformation levels, the practical measurement of memory savings on the GPU is acceptable and supports the main idea behind FAdamWav.

From the experiments, it has been observed that considerable memory savings are achieved with one or two wavelet transformation levels. However, applying more than two transformation levels is not recommended. In fact, Experiment 4 shows that using three levels of the wavelet transform significantly increases execution time without reducing the amount of GPU memory.

In Experiment 2, the filters are first calculated based on the parameters, and then the DWT is applied. It is important to note that these parameters can be adjusted during training. Specifically, the parameter selection comes from a finite set of values in the range $(0,2\pi )$. However, the filters can be modified during execution.

Our experiments only consider small models. However, it is reasonable to expect that the main idea of FAdamWav applies to more complex models such as Large Language Models (LLMs) or Convolutional Neural Networks (CNNs), which require a lot of memory. LLMs are trained on massive amounts of text data, such as ChatGPT [35], Gemini [36], or LLaMA [37], and represent a memory challenge for training. CNNs [38] are models designed to process grid-like data, such as images, for tasks like image classification, object detection, and facial recognition. When training a CNN, memory is consumed not only by the parameters but also by the gradients. Thus, they also represents a challenge for training.

In this sense, memory reduction is still an open research area, and we contribute with FAdamWav.

Our study reveals a trade-off between the execution time, the neural network accuracy, the memory reduction, and the number k of transformation levels. The experiments demonstrate this relationship and encourage further research.

Finally, based on the previous discussion, we conclude that is possible to reduce the memory required by Adam’s gradients while maintaining a comparable performance to non-wavelet versions. Additionally, different parametric filters modify the training performance of a neural network, and there exists an optimal number of transformation levels that promotes the memory saving, good execution time and competitive accuracy.

Future work will focus on studying the automatic adaptation of parametric filters, the fractional order derivative, and the number of wavelet transformation levels.