Next Article in Journal
GRU-SVM Based Threat Detection in Cognitive Radio Network
Previous Article in Journal
Smartphones and Threshold-Based Monitoring Methods Effectively Detect Falls Remotely: A Systematic Review
Previous Article in Special Issue
Visual Relationship Detection with Multimodal Fusion and Reasoning
 
 
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
This is an early access version, the complete PDF, HTML, and XML versions will be available soon.
Article

Less Is More: Adaptive Trainable Gradient Dropout for Deep Neural Networks

Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH), 57001 Thessaloniki, Greece
*
Author to whom correspondence should be addressed.
Sensors 2023, 23(3), 1325; https://doi.org/10.3390/s23031325
Received: 12 December 2022 / Revised: 17 January 2023 / Accepted: 17 January 2023 / Published: 24 January 2023

Abstract

The undeniable computational power of artificial neural networks has granted the scientific community the ability to exploit the available data in ways previously inconceivable. However, deep neural networks require an overwhelming quantity of data in order to interpret the underlying connections between them, and therefore, be able to complete the specific task that they have been assigned to. Feeding a deep neural network with vast amounts of data usually ensures efficiency, but may, however, harm the network’s ability to generalize. To tackle this, numerous regularization techniques have been proposed, with dropout being one of the most dominant. This paper proposes a selective gradient dropout method, which, instead of relying on dropping random weights, learns to freeze the training process of specific connections, thereby increasing the overall network’s sparsity in an adaptive manner, by driving it to utilize more salient weights. The experimental results show that the produced sparse network outperforms the baseline on numerous image classification datasets, and additionally, the yielded results occurred after significantly less training epochs.
Keywords: adaptive dropout; gradient dropout; gradient freezing; trainable dropout adaptive dropout; gradient dropout; gradient freezing; trainable dropout

Share and Cite

MDPI and ACS Style

Avgerinos, C.; Vretos, N.; Daras, P. Less Is More: Adaptive Trainable Gradient Dropout for Deep Neural Networks. Sensors 2023, 23, 1325. https://doi.org/10.3390/s23031325

AMA Style

Avgerinos C, Vretos N, Daras P. Less Is More: Adaptive Trainable Gradient Dropout for Deep Neural Networks. Sensors. 2023; 23(3):1325. https://doi.org/10.3390/s23031325

Chicago/Turabian Style

Avgerinos, Christos, Nicholas Vretos, and Petros Daras. 2023. "Less Is More: Adaptive Trainable Gradient Dropout for Deep Neural Networks" Sensors 23, no. 3: 1325. https://doi.org/10.3390/s23031325

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop