Training Neural Networks by Time-Fractional Gradient Descent

Motivated by the weighted averaging method for training neural networks, we study the time-fractional gradient descent (TFGD) method based on the time-fractional gradient flow and explore the influence of memory dependence on neural network training. The TFGD algorithm in this paper is studied via theoretical derivations and neural network training experiments. Compared with the common gradient descent (GD) algorithm, the optimization effect of the time-fractional gradient descent algorithm is significant when the value of fractional $\alpha$ is close to 1, under the condition of appropriate learning rate $\eta$. The comparison is extended to experiments on the MNIST dataset with various learning rates. It is verified that the TFGD has potential advantages when the fractional $\alpha$ nears 0.95∼0.99. This suggests that the memory dependence can improve training performance of neural networks.