Next Article in Journal
A Simplified Method to Predict the Heat Release Rate of Industrial Nitrocellulose Materials
Previous Article in Journal
Roadmap Modeling and Assessment Approach for Defense Technology System of Systems
Article Menu
Issue 6 (June) cover image

Export Article

Open AccessArticle
Appl. Sci. 2018, 8(6), 909;

Image Captioning with Word Gate and Adaptive Self-Critical Learning

1,2,‡,* , 3,‡
Information Security Center, State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China
National Engineering Laboratory for Disaster Backup and Recovery, Beijing University of Posts and Telecommunications, Beijing 100876, China
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
Current address: School of Cyberspace Security, Beijing University of Posts and Telecommunications, P.O.Box 145, Haidian District, Beijing 100876, China.
These authors contributed equally to this work.
Author to whom correspondence should be addressed.
Received: 25 April 2018 / Revised: 24 May 2018 / Accepted: 28 May 2018 / Published: 1 June 2018
(This article belongs to the Section Computer Science and Electrical Engineering)
Full-Text   |   PDF [3313 KB, uploaded 1 June 2018]   |  


Although the policy-gradient methods for reinforcement learning have shown significant improvement in image captioning, how to achieve high performance during the reinforcement optimizing process is still not a simple task. There are at least two difficulties: (1) The large size of vocabulary leads to a large action space, which makes it difficult for the model to accurately predict the current word. (2) The large variance of gradient estimation in reinforcement learning usually causes severe instabilities in the training process. In this paper, we propose two innovations to boost the performance of self-critical sequence training (SCST). First, we modify the standard long short-term memory (LSTM)based decoder by introducing a gate function to reduce the search scope of the vocabulary for any given image, which is termed the word gate decoder. Second, instead of only considering current maximum actions greedily, we propose a stabilized gradient estimation method whose gradient variance is controlled by the difference between the sampling reward from the current model and the expectation of the historical reward. We conducted extensive experiments, and results showed that our method could accelerate the training process and increase the prediction accuracy. Our method was validated on MS COCO datasets and yielded state-of-the-art performance. View Full-Text
Keywords: image caption; image understanding; deep learning; computer vision image caption; image understanding; deep learning; computer vision

Figure 1

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).

Share & Cite This Article

MDPI and ACS Style

Zhu, X.; Li, L.; Liu, J.; Guo, L.; Fang, Z.; Peng, H.; Niu, X. Image Captioning with Word Gate and Adaptive Self-Critical Learning. Appl. Sci. 2018, 8, 909.

Show more citation formats Show less citations formats

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Related Articles

Article Metrics

Article Access Statistics



[Return to top]
Appl. Sci. EISSN 2076-3417 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top