Next Article in Journal
Using Allan Variance Based Semi-Parameter Model to Calibrate Pointing Errors of Alt-az Telescopes
Previous Article in Journal
Dimensioning Models of Optical WDM Rings in Xhaul Access Architectures for the Transport of Ethernet/CPRI Traffic
Open AccessArticle

Self-Interaction Attention Mechanism-Based Text Representation for Document Classification

Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha 410073, China
Author to whom correspondence should be addressed.
Co-first authors of this article.
Appl. Sci. 2018, 8(4), 613;
Received: 17 March 2018 / Revised: 30 March 2018 / Accepted: 10 April 2018 / Published: 12 April 2018
Document classification has a broad application in the field of sentiment classification, document ranking and topic labeling, etc. Previous neural network-based work has mainly focused on investigating a so-called forward implication, i.e., the preceding text segments are taken as the context of the following text segments when generating the text representation. Such a scenario typically ignores the fact that the semantics of a document are a product of the mutual implication of all text segments in a document. Thus, in this paper, we introduce a concept of interaction and propose a text representation model with Self-interaction Attention Mechanism (TextSAM) for document classification. In particular, we design three aggregated strategies to integrate the interaction into a hierarchical architecture for document classification, i.e., averaging the interaction, maximizing the interaction and adding one more attention layer on the interaction, which leads to three models, i.e., TextSAMAVE, TextSAMMAX and TextSAMATT, respectively. Our comprehensive experimental results on two public datasets, i.e., Yelp 2016 and Amazon Reviews (Electronics), show that our proposals can significantly outperform the state-of-the-art neural-based baselines for document classification, presenting a general improvement in terms of accuracy ranging from 5.97% to 14.05% against the best baseline. Furthermore, we find that our proposals with a self-interaction attention mechanism can obviously alleviate the impact brought by the increase of sentence number as the relative improvement of our proposals against the baselines are enlarged when the sentence number increases. View Full-Text
Keywords: interaction representation; attention mechanism; document classification; hierarchical architecture interaction representation; attention mechanism; document classification; hierarchical architecture
Show Figures

Figure 1

MDPI and ACS Style

Zheng, J.; Cai, F.; Shao, T.; Chen, H. Self-Interaction Attention Mechanism-Based Text Representation for Document Classification. Appl. Sci. 2018, 8, 613.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

Back to TopTop