Skip to Content
You are currently on the new version of our website. Access the old version .
AxiomsAxioms
  • Article
  • Open Access

26 September 2022

11 Pages

Training Neural Networks by Time-Fractional Gradient Descent

and
School of Mathematics and Statistics, Guizhou University, Guiyang 550025, China
*
Author to whom correspondence should be addressed.
This article belongs to the Special Issue Impulsive, Delay and Fractional Order Systems

Abstract

Motivated by the weighted averaging method for training neural networks, we study the time-fractional gradient descent (TFGD) method based on the time-fractional gradient flow and explore the influence of memory dependence on neural network training. The TFGD algorithm in this paper is studied via theoretical derivations and neural network training experiments. Compared with the common gradient descent (GD) algorithm, the optimization effect of the time-fractional gradient descent algorithm is significant when the value of fractional α is close to 1, under the condition of appropriate learning rate η. The comparison is extended to experiments on the MNIST dataset with various learning rates. It is verified that the TFGD has potential advantages when the fractional α nears 0.95∼0.99. This suggests that the memory dependence can improve training performance of neural networks.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.