Revisiting Dropout: Escaping Pressure for Training Neural Networks with Multiple Costs
1
Electrical Engineering and Computer Science Department & Artificial Intelligence Graduate School, Gwangju Institute of Science and Technology (GIST), 123, Cheomdangwagi-ro, Buk-gu, Gwangju 61005, Korea
2
Lawrence Livermore National Laboratory, 7000 East Avenue, Livermore, CA 94550, USA
3
Electronics and Telecommunications Research Institute (ETRI), 218, Gajeong-ro, Yuseong-gu, Daejeon 34129, Korea
4
Computer Science and Engineering Department, Jeonbuk National University, Baekje-daero, Deokjin-gu, Jeonju 54896, Korea
*
Authors to whom correspondence should be addressed.
Academic Editor: Daniel Gutiérrez Reina
Electronics 2021, 10(9), 989; https://doi.org/10.3390/electronics10090989
Received: 17 March 2021
/
Revised: 12 April 2021
/
Accepted: 19 April 2021
/
Published: 21 April 2021
(This article belongs to the Section Artificial Intelligence)
A common approach to jointly learn multiple tasks with a shared structure is to optimize the model with a combined landscape of multiple sub-costs. However, gradients derived from each sub-cost often conflicts in cost plateaus, resulting in a subpar optimum. In this work, we shed light on such gradient conflict challenges and suggest a solution named Cost-Out, which randomly drops the sub-costs for each iteration. We provide the theoretical and empirical evidence of the existence of escaping pressure induced by the Cost-Out mechanism. While simple, the empirical results indicate that the proposed method can enhance the performance of multi-task learning problems, including two-digit image classification sampled from MNIST dataset and machine translation tasks for English from and to French, Spanish, and German WMT14 datasets.
View Full-Text
Keywords:
multitask learning; gradient conflict; Cost-Out; escaping pressure; dropout
▼
Show Figures
Figure 1
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited
MDPI and ACS Style
Woo, S.; Kim, K.; Noh, J.; Shin, J.-H.; Na, S.-H. Revisiting Dropout: Escaping Pressure for Training Neural Networks with Multiple Costs. Electronics 2021, 10, 989. https://doi.org/10.3390/electronics10090989
AMA Style
Woo S, Kim K, Noh J, Shin J-H, Na S-H. Revisiting Dropout: Escaping Pressure for Training Neural Networks with Multiple Costs. Electronics. 2021; 10(9):989. https://doi.org/10.3390/electronics10090989
Chicago/Turabian StyleWoo, Sangmin, Kangil Kim, Junhyug Noh, Jong-Hun Shin, and Seung-Hoon Na. 2021. "Revisiting Dropout: Escaping Pressure for Training Neural Networks with Multiple Costs" Electronics 10, no. 9: 989. https://doi.org/10.3390/electronics10090989
Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.
