This is an early access version, the complete PDF, HTML, and XML versions will be available soon.
Less Is More: Adaptive Trainable Gradient Dropout for Deep Neural Networks
Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH), 57001 Thessaloniki, Greece
Author to whom correspondence should be addressed.
Received: 12 December 2022
Revised: 17 January 2023
Accepted: 17 January 2023
Published: 24 January 2023
The undeniable computational power of artificial neural networks has granted the scientific community the ability to exploit the available data in ways previously inconceivable. However, deep neural networks require an overwhelming quantity of data in order to interpret the underlying connections between them, and therefore, be able to complete the specific task that they have been assigned to. Feeding a deep neural network with vast amounts of data usually ensures efficiency, but may, however, harm the network’s ability to generalize. To tackle this, numerous regularization techniques have been proposed, with dropout being one of the most dominant. This paper proposes a selective gradient dropout method, which, instead of relying on dropping random weights, learns to freeze the training process of specific connections, thereby increasing the overall network’s sparsity in an adaptive manner, by driving it to utilize more salient weights. The experimental results show that the produced sparse network outperforms the baseline on numerous image classification datasets, and additionally, the yielded results occurred after significantly less training epochs.
Share and Cite
MDPI and ACS Style
Avgerinos, C.; Vretos, N.; Daras, P. Less Is More: Adaptive Trainable Gradient Dropout for Deep Neural Networks. Sensors 2023, 23, 1325.
Avgerinos C, Vretos N, Daras P. Less Is More: Adaptive Trainable Gradient Dropout for Deep Neural Networks. Sensors. 2023; 23(3):1325.
Avgerinos, Christos, Nicholas Vretos, and Petros Daras. 2023. "Less Is More: Adaptive Trainable Gradient Dropout for Deep Neural Networks" Sensors 23, no. 3: 1325.
Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here
Article Access Statistics
For more information on the journal statistics, click here
Multiple requests from the same IP address are counted as one view.