Next Article in Journal
Efficiently Estimating Joining Cost of Subqueries in Regular Path Queries
Previous Article in Journal
Transparent All-Oxide Hybrid NiO:N/TiO2 Heterostructure for Optoelectronic Applications
Open AccessArticle

Revisiting Dropout: Escaping Pressure for Training Neural Networks with Multiple Costs

1
Electrical Engineering and Computer Science Department & Artificial Intelligence Graduate School, Gwangju Institute of Science and Technology (GIST), 123, Cheomdangwagi-ro, Buk-gu, Gwangju 61005, Korea
2
Lawrence Livermore National Laboratory, 7000 East Avenue, Livermore, CA 94550, USA
3
Electronics and Telecommunications Research Institute (ETRI), 218, Gajeong-ro, Yuseong-gu, Daejeon 34129, Korea
4
Computer Science and Engineering Department, Jeonbuk National University, Baekje-daero, Deokjin-gu, Jeonju 54896, Korea
*
Authors to whom correspondence should be addressed.
Academic Editor: Daniel Gutiérrez Reina
Electronics 2021, 10(9), 989; https://doi.org/10.3390/electronics10090989
Received: 17 March 2021 / Revised: 12 April 2021 / Accepted: 19 April 2021 / Published: 21 April 2021
(This article belongs to the Section Artificial Intelligence)
A common approach to jointly learn multiple tasks with a shared structure is to optimize the model with a combined landscape of multiple sub-costs. However, gradients derived from each sub-cost often conflicts in cost plateaus, resulting in a subpar optimum. In this work, we shed light on such gradient conflict challenges and suggest a solution named Cost-Out, which randomly drops the sub-costs for each iteration. We provide the theoretical and empirical evidence of the existence of escaping pressure induced by the Cost-Out mechanism. While simple, the empirical results indicate that the proposed method can enhance the performance of multi-task learning problems, including two-digit image classification sampled from MNIST dataset and machine translation tasks for English from and to French, Spanish, and German WMT14 datasets. View Full-Text
Keywords: multitask learning; gradient conflict; Cost-Out; escaping pressure; dropout multitask learning; gradient conflict; Cost-Out; escaping pressure; dropout
Show Figures

Figure 1

MDPI and ACS Style

Woo, S.; Kim, K.; Noh, J.; Shin, J.-H.; Na, S.-H. Revisiting Dropout: Escaping Pressure for Training Neural Networks with Multiple Costs. Electronics 2021, 10, 989. https://doi.org/10.3390/electronics10090989

AMA Style

Woo S, Kim K, Noh J, Shin J-H, Na S-H. Revisiting Dropout: Escaping Pressure for Training Neural Networks with Multiple Costs. Electronics. 2021; 10(9):989. https://doi.org/10.3390/electronics10090989

Chicago/Turabian Style

Woo, Sangmin; Kim, Kangil; Noh, Junhyug; Shin, Jong-Hun; Na, Seung-Hoon. 2021. "Revisiting Dropout: Escaping Pressure for Training Neural Networks with Multiple Costs" Electronics 10, no. 9: 989. https://doi.org/10.3390/electronics10090989

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Search more from Scilit
 
Search
Back to TopTop