Next Article in Journal
Interference Alignment Inspired Opportunistic Communications in Multi-Cluster MIMO Networks with Wireless Power Transfer
Next Article in Special Issue
Ranking Algorithms for Word Ordering in Surface Realization
Previous Article in Journal
Geometric Regularization of Local Activations for Knowledge Transfer in Convolutional Neural Networks
Previous Article in Special Issue
Semantic Systematicity in Connectionist Language Production
Article

Goal-Driven Visual Question Generation from Radiology Images

U.S. National Library of Medicine, National Institutes of Health, Bethesda, MD 20894, USA
*
Authors to whom correspondence should be addressed.
Academic Editors: Valerio Basile and Alessandro Mazzei
Information 2021, 12(8), 334; https://doi.org/10.3390/info12080334
Received: 15 July 2021 / Revised: 14 August 2021 / Accepted: 16 August 2021 / Published: 20 August 2021
(This article belongs to the Special Issue Neural Natural Language Generation)
Visual Question Generation (VQG) from images is a rising research topic in both fields of natural language processing and computer vision. Although there are some recent efforts towards generating questions from images in the open domain, the VQG task in the medical domain has not been well-studied so far due to the lack of labeled data. In this paper, we introduce a goal-driven VQG approach for radiology images called VQGRaD that generates questions targeting specific image aspects such as modality and abnormality. In particular, we study generating natural language questions based on the visual content of the image and on additional information such as the image caption and the question category. VQGRaD encodes the dense vectors of different inputs into two latent spaces, which allows generating, for a specific question category, relevant questions about the images, with or without their captions. We also explore the impact of domain knowledge incorporation (e.g., medical entities and semantic types) and data augmentation techniques on visual question generation in the medical domain. Experiments performed on the VQA-RAD dataset of clinical visual questions showed that VQGRaD achieves 61.86% BLEU score and outperforms strong baselines. We also performed a blinded human evaluation of the grammaticality, fluency, and relevance of the generated questions. The human evaluation demonstrated the better quality of VQGRaD outputs and showed that incorporating medical entities improves the quality of the generated questions. Using the test data and evaluation process of the ImageCLEF 2020 VQA-Med challenge, we found that relying on the proposed data augmentation technique to generate new training samples by applying different kinds of transformations, can mitigate the lack of data, avoid overfitting, and bring a substantial improvement in medical VQG. View Full-Text
Keywords: visual question generation; visual question answering; variational autoencoders; radiology images; domain knowledge; unified medical language system; data augmentation; computer vision; natural language processing; artificial intelligence; medical domain visual question generation; visual question answering; variational autoencoders; radiology images; domain knowledge; unified medical language system; data augmentation; computer vision; natural language processing; artificial intelligence; medical domain
Show Figures

Figure 1

MDPI and ACS Style

Sarrouti, M.; Ben Abacha, A.; Demner-Fushman, D. Goal-Driven Visual Question Generation from Radiology Images. Information 2021, 12, 334. https://doi.org/10.3390/info12080334

AMA Style

Sarrouti M, Ben Abacha A, Demner-Fushman D. Goal-Driven Visual Question Generation from Radiology Images. Information. 2021; 12(8):334. https://doi.org/10.3390/info12080334

Chicago/Turabian Style

Sarrouti, Mourad, Asma Ben Abacha, and Dina Demner-Fushman. 2021. "Goal-Driven Visual Question Generation from Radiology Images" Information 12, no. 8: 334. https://doi.org/10.3390/info12080334

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop