A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students
1
Graduate School of Advanced Science and Engineering, Waseda University, Tokyo 169-8050, Japan
2
National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba 305-8560, Japan
3
Graduate School of Information Science and Technology, The University of Tokyo, Tokyo 113-8654, Japan
4
Waseda Research Institute for Science and Engineering, Tokyo 169-8050, Japan
*
Author to whom correspondence should be addressed.
Academic Editors: Valentina Franzoni, Alfredo Milani and Giulio Biondi
Appl. Sci. 2021, 11(14), 6287; https://doi.org/10.3390/app11146287
Received: 10 June 2021
/
Revised: 2 July 2021
/
Accepted: 4 July 2021
/
Published: 7 July 2021
(This article belongs to the Special Issue Advances in Artificial Intelligence Learning Technologies)
Summaries of scientific publications enable readers to gain an overview of a large number of studies, but users’ preferences have not yet been explored. In this paper, we conduct two user studies (i.e., short- and long-term studies) where Japanese university students read summaries of English research articles that were either manually written or automatically generated using text summarization and/or machine translation. In the short-term experiment, subjects compared and evaluated the two types of summaries of the same article. We analyze the characteristics in the generated summaries that readers regard as important, such as content richness and simplicity. The experimental results show that subjects are mainly judged based on four criteria, including content richness, simplicity, fluency, and format. In the long-term experiment, subjects read 50 summaries and answered whether they would like to read the original papers after reading the summaries. We discuss the characteristics in the summaries that readers tend to use to determine whether to read the papers, such as topic, methods, and results. The comments from subjects indicate that specific components of scientific publications, including research topics and methods, are important to judge whether to read or not. Our study provides insights to enhance the effectiveness of automatic summarization of scientific publications.
View Full-Text
Keywords:
user evaluation; scientific publication; academic activity analysis; language processing application
▼
Show Figures
Figure 1
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
MDPI and ACS Style
Yamamoto, S.; Suzuki, R.; Fukusato, T.; Kataoka, H.; Morishima, S. A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students. Appl. Sci. 2021, 11, 6287. https://doi.org/10.3390/app11146287
AMA Style
Yamamoto S, Suzuki R, Fukusato T, Kataoka H, Morishima S. A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students. Applied Sciences. 2021; 11(14):6287. https://doi.org/10.3390/app11146287
Chicago/Turabian StyleYamamoto, Shintaro, Ryota Suzuki, Tsukasa Fukusato, Hirokatsu Kataoka, and Shigeo Morishima. 2021. "A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students" Applied Sciences 11, no. 14: 6287. https://doi.org/10.3390/app11146287
Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.
