Attentional Interactive Encoder Network Focused on Aspect for Sentiment Classification

Abstract

Aspect-based sentiment analysis (ABSA) plays a significant role in the field of big data and aims to distinguish the sentiment polarity of specific aspects in given sentences; however, the previous works on ABSA had two limitations. They mainly considered semantic features, rather than syntactic dependency features, and paid too much attention to the context words, while ignoring the high-level interaction of multiple representations of aspects themselves. To cope with these limitations, we propose a new method based on the graph convolutional network (GCN) and the multi-head attention mechanism, called the attention interactive encoder network (AIEN). The GCN was used to obtain the syntactic information that has the greatest syntactic impact on the aspect based on the syntax dependency tree. The multi-head attention mechanism can not only obtain the context-aware information of the given aspects, but also the interaction information between multiple representations of the aspect itself. The high-level information generated by the interaction of multi-dimensional features can produce a stronger representation ability for the aspect. Our experiments with the proposed model on five benchmark datasets showed that our model outperformed other works significantly. The experimental results further demonstrated the feasibility and applicability of our proposed model in the ABSA task.

1. Introduction

As an emerging technology, big data analysis aims to collect massive data, analyze and mine the hidden relationship between datasets, and establish corresponding data [1,2]. Due to the rapid development of AI technology, the use of deep learning methods for data analysis has become the norm in industry and academia [3]. In natural language processing, sentiment analysis is a common data analysis task, widely used in various text and voice services [4,5,6]. Among them, aspect-based (also known as aspect-level) sentiment analysis (ABSA) is a fine-grained sentiment analysis task for a given aspect [7]. It determines the relative sentiment polarity of each aspect of the sentence, rather than outputting a single sentiment polarity for the whole sentence. The sentiment polarity of a particular aspect item is closely related to its contextual words and syntactic dependencies, so the polarity of each aspect in a sentence is usually unique. For example, in the sentence “The pizza is delicious, but the service is terrible.”, pizza and service have different emotion polarity.

Early work applied rule-based or statistical methods to ABSA tasks with obvious manual characteristics [8]. Lately, machine learning and neural networks have been broadly utilized in ABSA tasks [9]. The mainstream is to develop a neural network structure that can learn the features of aspects and context words such as semantic features. Because long-term and short-term memory (LSTM) can extract the sequential characterizations of sentences [10] and the attention mechanism can pay more attention to the important information of context words for a given aspect and reduce the interference of irrelevant information [11], these two advanced technologies are widely used in ASBA tasks. However, the attention mechanism has some limitations. Generally speaking, it is impossible to obtain the complex syntactic dependency between aspect and context words, because it only outputs their relative importance. For example, from “Its size is ideal and its weight is acceptable”, the attention mechanism may misinterpret “acceptable” as a key contextual word describing size. To capture the relationship between aspect and context, recent research tried to obtain the syntactical dependency of these two contents in ABSA based on a graph convolutional network (GCN) [12,13]. Nevertheless, these works focused more on context words and less on interactive information between context words and aspects.

In short, the current research on ABSA has two limitations. First, previous works mainly focused on semantic features rather than syntactic dependency features [14]. They mainly focused on context-aware features without considering aspects. At the same time, they also ignored some syntactical dependency of aspect related to context words, which may lead to their inability to capture the information given by the most-applicable context words about determining the sentiment polarity of a particular aspect. In addition, some work focused on the intuitive information between context words and aspects, but ignored the interaction between features of multiple representations of aspects [12,13]. However, these multiple representations of aspects contain rich information, such as semantics, syntax, context-aware features, etc. Through the interaction between these contents containing comprehensive information to obtain advanced features, various aspects of information can be synthesized and the representation of aspects can be improved, so as to more effectively describe the emotion polarity of aspects.

Hence, we designed an attentional interactive encoder network (AIEN) to tackle these two problems. In the AIEN, how to explore the semantic and syntactic dependencies centered on aspect itself will become the main concern in this paper. Firstly, the semantic features of context and aspect in sentences are captured by LSTM, respectively. Then, we used the GCN to obtain the syntactic dependency features from the graph constructed according to the syntactic dependency tree. A simple schematic of syntactical dependency analysis is shown in Figure 1.

Interactive features can integrate multiple feature capabilities to obtain a better representation. Therefore, after acquiring the multi-dimensional representation features of aspect and context words, we need to acquire the high-level interaction features centered on the aspect. We propose a strategy based on the attention mechanism to obtain interactive information on aspects according to various features. Considering the multiple representations of aspects in semantics and syntax and the influence of context words on aspects, we obtained two types of high-level interaction features of aspects based on the multi-head attention mechanism, including the interaction between multiple representations of aspects and the interaction between aspects and context words. These interactive features improved the basic characterization ability of aspects.

Our contributions are as follows:

1. We endeavored to investigate multiple representations of aspects, including the semantic features of the aspect word itself, as well as the semantic and syntactic dependency features in the whole sentence.

2. We propose a high-level interactive feature extraction strategy focusing on aspect by the attention mechanism. These generated high-level interaction features consider both semantic characteristics and syntactical dependence in the meantime.

3. The extensive experimental results showed that our model performed better than the state-of-the-art models. Our ablation experiments additionally confirmed the significance of semantic features and syntactic dependence information; meanwhile, they also demonstrated the effectiveness of interaction features focused on aspect in the ABSA task.

The rest of this paper is organized as follows. Section 2 reviews some existing works related to our work. Section 3 presents the architecture of our model the AIEN. Section 4 shows the experimental results and analyzes them. Finally, Section 5 concludes this work.

2. Related Works

2.1. Aspect-Based Sentiment Analysis

As a fine-grained sentiment classification task, ABSA has attracted the attention of researchers and has gradually become an important branch in the field of sentiment analysis. In early studies, researchers mainly extracted features of the specific aspect through rule-based and statistics methods. Most of these methods are manual and labor-intensive [15]. With the emergence of deep learning (DL), many scholars have turned to explore ABSA tasks by introducing neural networks. In the beginning, researchers only extracted the semantics of the text through LSTM, not considering the relationship between aspect and context [9,16]. Later research realized the importance of this relationship between aspect and context words and proposed to use LSTM and the attention mechanism to establish the sentiment polarity of aspects considering aspect and context. Because of the excellent performance of attention in the natural language processing (NLP) and computer vision (CV) tasks, many works have also tried to enhance the important information of context words related to aspect through the attention method in the ABSA task [17,18,19]. While enjoying the benefits brought by the attention mechanism, the researchers also considered integrating other methods to improve the model effect of ABSA tasks, such as feature interaction and regularization. Reference [20] proposed an interactive attention network (IAN) to interactively learn contextual and aspect attention and generate representations for aspect and context, respectively. Reference [14] used the attention encoder network to model the relationship between the context and the target and introduced label smoothing regularization to improve the results. These methods simply model the semantic information of aspect and context words. However, they lack information about the syntactic dependencies. Thus, this may result in irrelevant contexts being involved in determining the sentiment polarity of aspect.

2.2. Graph Convolutional Network

The graph convolutional neural network (GCN) proposed by [21] can be effectively used to process graph-structured data containing rich inter-node relationship information. The GCN was introduced into computer vision processing and applied to the tasks of image segmentation and video detection [22,23]. The GCN was also applied to social network mining and recommendation system, to build graphs through the relationship between users or users and items [24]. Furthermore, the GCN was extended to NLP tasks such as text classification [25], machine translation [26], and relation extraction [27]. In recent years, there have been some works that have attempted to introduce the GCN into sentiment analysis. Reference [12] first built a GCN on the syntactic dependence tree to use syntactic information to solve related syntactic constraints and long-distance word dependence. Reference [28] combined the attention mechanism and the GCN to capture the syntactic dependence between different aspects of a sentence. Reference [13] proposed a model composed of the attention mechanism and the graph convolutional network built on sentence dependency trees. Reference [29] proposed a directed GCN, which performs joint extraction and sentiment analysis by encoding syntactic information. Although these methods attempt to obtain syntactic dependency features through the GCN, they do not consider the feature interactions of aspect and context words at the same time. In this paper, we propose a novel method called the AIEN to obtain comprehensive interactive features focusing on the aspect-based GCN and attention mechanism.

3. Methodology

The architecture of our developed AIEN model is shown in Figure 2. In the figure, “Embedding” represents GloVe embedding or pre-trained BERT embedding; “Hidden states” represents the Bi-LSTM; “GCN Layer” represents the layers in the GCN; “Mask Layer” denotes the layer of the aspect-specific masking technique; “MHA” refers to multi-head attention.

3.1. Embedding

Given a sentence $s=\left\{w_1^s,w_2^s,\dots ,w_{\tau }^s,w_{(\tau +1)}^s,\dots \right.,\left.w_{(\tau +m)}^s,\dots ,w_n^s\right\}$ of n words including corresponding m words’ aspect beginning from the $(\tau +1)$ to $(\tau +m)$ token, through word embedding, we map each word to a low-dimensional real-valued vector space. The embedding matrix is denoted as $E\in {\mathbb{R}}^{d\times \left|V\right|}$, where d is the embedding dimension of word vectors and $\left|V\right|$ is the size of the vocabulary. In our work, the pre-trained word embedding of GloVe [30] and BERT [31] is used to initialize word embeddings.

Considering the BERT embedding layer, the network architecture of BERT uses a multi-layer transformer encoder structure; here, we used the most-basic BERT model, which has 12 transformer layers. The training of the BERT model was divided into two steps: pre-training and fine-tuning. After pre-training, BERT returns two feature vectors, which contain the context information extracted for aspect and context, respectively. Then, we fine-tuned the model. To simplify, we first added a dropout layer after the output of the model and then added a linear layer. After word embedding, we encoded an n-word sentence and an m-word target embedding vector $c=\left\{e_1^c,e_2^c,e_3^c,\dots ,e_n^c\right\}$ and $t=\left\{e_1^t,e_2^t,e_3^t,\dots ,e_m^t\right\}$, respectively.

3.2. Semantic Information Encoding

3.2.1. Bidirectional LSTM

We employed bidirectional LSTM (Bi-LSTM) in the next layer of the embedding to capture the context information of each word, which can effectively use the previous words and future words of contextual information and then summarize the information in two directions to obtain word features.

The word embedding vectors including context and target words are sent into Bi-LSTM, respectively. Then, the Bi-LSTM produces the hidden state vector for contexts and the target, respectively, as follows:

$$\begin{array}{c}\hfill H_c=\left\{h_1^c,h_2^c,h_3^c,\dots ,h_n^c\right\}\end{array}$$

(1a)

$$\begin{array}{c}\hfill H_t=\left\{h_1^t,h_2^t,h_3^t,\dots ,h_n^t\right\}\end{array}$$

(1b)

where $\left(h_i^c,h_i^t\right)\in {\mathbb{R}}^{2d_h}$ represents the hidden state vector at time i in Bi-LSTM and $d_h$ is the dimension of the hidden state vector output by undirected LSTM.

3.2.2. Aspect-Specific Masking

To obtain aspect features’ focusing, we designed a masking mechanism that filters out non-aspect words and keeps solely aspect-specific features:

$$\begin{array}{c}{h}_t^L=01\le t<\tau +1,\tau +m<t\le n\end{array}$$

(2)

$$\begin{array}{c}{H}_{mask}=\left\{0,\dots ,h_{\tau },\dots ,h_{\tau +m},\dots ,0\right\}\end{array}$$

(3)

As shown in Equation (2), $H_{mask}$ represents the semantic information filtered through the mask mechanism. In our work, we obtained the context-perceived features’ focusing aspect through a masking mechanism. The masked information focuses on aspects. At this time, aspects have rich contextual semantic information.

3.3. Syntactic Information Encoding

3.3.1. Position Encoding

Empirically, we found that the polarity of a specific aspect is more easily affected by the context words that are closer to the aspect. Therefore, we employed position encoding to simulate these normal rules before feeding into a successive GCN layer. Formally, given an aspect W, which is indexed from $\tau$ to $\tau +m$, the relative distance weight of the i-th word is defined as follows:

$$q_i=\left\{\begin{array}{cc}1-\frac{\tau +1-i}{n}\hfill & 1\le i<\tau +1\hfill \\ 1\hfill & \tau +1\le i\le \tau +m\hfill \\ 1-\frac{i-\tau -m}{n}\hfill & \tau +m<i\le n\hfill \end{array}\right.$$

(4)

where $q_i\in R$ is the position weight to the i-th token. In Equation (4), $q_i=1$ means that the word in the sentence is an aspect word. That indicates that the position distance weight of the aspect is 1, and then, the weight is gradually reduced from the aspect to the sides. Finally, we can obtain the position-aware representation with position information:

$$H=\left\{{\mathrm{h}}_1^L,\cdots ,{\mathrm{h}}_{\tau +1}^L,\cdots ,{\mathrm{h}}_{\tau +m}^L,\cdots ,{\mathrm{h}}_n^L\right\}$$

(5)

3.3.2. Graph Convolutional Network

The graph convolutional network (GCN) can effectively obtain the correlation of the graph through the information transfer between the nodes of the graph. Even if it is not trained and completely uses the randomly initialized parameter W, the features extracted by the GCN are excellent. Due to the GCN capturing the connectivity of the graph through information propagation between the nodes of the graph, we can use the GCN to solve the sentiment dependency between aspect and context of the ABSA problem. We used the GCN on the syntactic dependency tree of sentences and manually set each word to be adjacent to itself according to the idea of the GCN, that is the diagonal elements of the adjacency matrix are ones. The representation of each node is updated with the graph convolutional operation with a normalization factor, which is shown below:

$$h_i^{\left(k\right)}=f\left(W^{\left(k\right)}·\frac{{\sum }_{u\in \mathcal{N}\left(i\right)}{h}_u^{(k-1)}}{\left|\mathcal{N}\right(i\left)\right|}+B·h_i^{(k-1)}\right)$$

(6)

where $h_i^{\left(k\right)}$ is node v’s embedding at step k and $h_i^{\left(0\right)}$ is node v’s initial embedding specifically. The embedding of a neighbor of node v is $h_u^{(k-1)}$. Apparently, $\frac{{\sum }_{u\in \mathcal{N}\left(i\right)}{h}_u^{(k-1)}}{\left|\mathcal{N}\right(i\left)\right|}$ is the mean of v’s neighbor’s embeddings at step $k-1$. For each step k, the function f and matrices W and B are potentially learnable parameters and shared across all nodes. Especially, $h_i^0=X$, which is a feature description $x_i$ for every node i, summarized in an $N\times D$ feature matrix X (N: number of nodes, D: number of input features). $h_i^L=Z$, which is a node-level output Z (an $N\times F$ feature matrix, where F is the number of output features per node), and L is the number of layers.

In our work, we used a two-layer GCN to exploit text information, and the calculation can be indicated as follows:

$$\begin{array}{c}\hfill Z^{\left(0\right)}=Relu\left(\tilde{A}X{W}^{\left(0\right)}\right)\end{array}$$

(7a)

$$\begin{array}{c}\hfill Z^{\left(1\right)}=softmax\left(\tilde{A}{Z}^{\left(0\right)}{W}^{\left(1\right)}\right)\end{array}$$

(7b)

where $Z^{\left(0\right)}$ represents the first layer and $Z^{\left(1\right)}$ represents the second layer. $Relu$ is a nonlinear activation function, which can effectively alleviate the overfitting problem. The final output is $H_g$:

$$H^g=softmax\left(\tilde{A}\left(Relu\left(\tilde{A}{X}^l{W}^{\left(0\right)}\right)\right)W^{\left(1\right)}\right)$$

(8)

Then, by analogy with the aspect semantic feature extraction of the context, we also applied the mask mechanism to obtain the syntactic dependency features of the aspect in the context. As shown in Equation (8), $H_g$ is the syntax-dependent feature of aspect.

3.4. Aspectwise Feature Interaction

3.4.1. Multi-Head Attention

Our model is based on multi-head attention (MHA), which can be performed in parallel subspaces. The main method of MHA is to divide the dimension of attention parameters (Query, Key, and Value) into multiple subspaces. After segmentation, each subspace can pay attention to different feature spaces to improve the performance of the model and, finally, put together the results of these subspaces. We define a Key sequence $k=\{k_1,k_2,\cdots ,k_n\}$ and a Query sequence $q=\{q_1,q_2,\cdots ,q_n\}$. Then, the mathematical calculation is as follows:

$$\mathrm{Attention}(k,q)=softmax\left(f_m(k,q)\right)k$$

(9)

where $f_m$ is the function utilized to calculate and learn the semantic relation between $q_j$ and $k_i$:

$$f_m\left(k_i,q_j\right)=tanh\left(\left[k_i;q_j\right]·W_a\right)$$

(10)

where $W_a\in R^{2d_{hid}}$ is the parameter learnable matrix. MHA is able to learn the scores of different $n_{head}$ in parallel spaces, and the parameters are not shared between these heads, because the values of q and k are continuously changing. The results of $n_{head}$ are connected and projected to the particular hidden dimension $d_{hid}$ by

$$\begin{array}{c}MHA=\left[o^1\oplus o^2\oplus \dots \oplus o^{n_{\mathrm{head}}}\right]·W_m\end{array}$$

(11)

$$\begin{array}{c}{o}^h={Attention}^h(k,q)\end{array}$$

(12)

where “⊕” represents the vector concatenation, $W_m\in R^{d_{hid}\times d_{hid}}$, $o^h=\left\{o_1^h,o_2^h,\dots ,o_m^h\right\}$ is the output of the h-th head’s attention, and $h\in \left[1,n_{\mathrm{head}}\right]$.

3.4.2. Aspect to Aspect Interaction

Through the above work, we obtained aspect-focused contextual features $H^{ga}$, semantic features $H^{ca}$, and syntactic features $H^{cg}$, as well as the semantic features $H^a$ of the aspect itself. Considering that the information contained in different representation forms of aspect is different, its high-level features are obtained through interaction. We realized the interaction through MHA, including the interaction of context-aware semantic features and syntactic features and the interaction between the aspect’s own features and context-aware aspect features, respectively. The calculation formulas of these three groups of focused aspect interaction characteristics are as follows:

$$\begin{array}{c}\hfill H^{ca}=\mathrm{MultiHead}\left(H^c,H^a\right)\end{array}$$

(13a)

$$\begin{array}{c}\hfill H^{ga}=\mathrm{MultiHead}\left(H^g,H^a\right)\end{array}$$

(13b)

$$\begin{array}{c}\hfill H^{cg}=\mathrm{MultiHead}\left(H^c,H^g\right)\end{array}$$

(13c)

where $H^{ca}=\{h_1^{ca},h_2^{ca},\dots ,h_n^{ca}\in R^{d_h\times n}\}$, $H^{ga}=\{h_1^{ga},h_2^{ga},\dots ,h_n^{ga}\in R^{d_h\times n}\}$, $H^{cg}=\{h_1^{cg},h_2^{cg},\dots ,h_n^{cg}\in R^{d_h\times n}\}$ and $d_h$ is the dimension of MHIA.

3.5. Output and Training

After obtaining the aspect-focused contextual features representation $h^{ga}$, semantic feature representation $h^{ca}$, and syntactic feature represent $h^{cg}$, we first obtained the average vector by adapting the averaging pooling method and then concatenated the aspect feature with the interactive feature between aspect and context, the interaction feature between aspect and GCN context, the interaction feature between context and GCN context, and the GCN context feature. The final feature representation u is as follows:

$$u=\left[h_{avg}^c\oplus h_{avg}^{ca}\oplus h_{avg}^{ga}\oplus h_{avg}^{cg}\oplus h_{avg}^a\right]$$

(14)

where “⊕” represents the vector concatenation and $h_{avg}$ stands for $\frac{{\sum }_{i=1}^n{h}_i}{n}$. Finally, the probability distribution of different target sentiment polarities is obtained through the output of the softmax layer, as follows:

$$\begin{array}{c}x={\tilde{W}}_u^T\tilde{u}+b_u\end{array}$$

(15)

$$\begin{array}{c}y=softmax\left(x\right)=\frac{exp\left(x\right)}{{\sum }_{k=1}^{C}exp\left(x\right)}\end{array}$$

(16)

Our model was trained by minimizing the cross-entropy with the L2-regularization term. For a given sentence, the loss function is represented by

$$Loss=-\sum _{(c,l)\in C}log{P}_l+\lambda {\parallel \Theta \parallel }_2$$

(17)

where C is the collection of datasets, l is the label, $P_l$ means the l-th element of P, $\Theta$ represents all of the trainable parameters, and $\lambda$ is the parameter of regularization.

4. Experiments

This section presents the experimental setup, including the dataset, evaluation metrics, and baseline, in which the effectiveness of our proposed model and its comparison with other models were shown. The outstanding features of the model for the ABSA task were highlighted by conducting an ablation experiment, parameter experiment, and case study on the model.

4.1. Experiment Setting

4.1.1. Datasets

To comprehensively evaluate the performance of the AIEN, we conducted experiments on five real datasets. The TWITTER dataset was originally constructed by [16] and contains Twitter posts, while the other four datasets (LAP14, REST14, REST15, REST16) were derived from SemEval 2014 Task 4 [32], SemEval 2015 Task 12 [33], and SemEval 2016 Task 5 [34]. These four datasets include two categories: laptop and restaurant. Each sample consists of the review sentences, aspects, and sentiment polarity towards the aspects. These datasets are labeled with three sentiment polarities: positive, neutral, and negative. For the fairness and rationality of the experiment, we used the same training set, validation set, and test set data distribution as all other baseline models.

Table 1. Statistics of datasets used in this paper.

Datasets		Pos	Neu	Neg	Total
TWITTER	Train	1561	3127	1560	6248
	Test	173	346	173	692
LAP14	Train	994	464	870	2328
	Test	341	169	128	638
REST14	Train	2164	637	807	3608
	Test	728	196	196	1120
REST15	Train	912	36	256	1204
	Test	326	34	182	542
REST16	Train	1240	69	439	1748
	Test	469	30	117	616

shows the number of training and test instances in each dataset.

4.1.2. Evaluation Metrics

The experimental results were obtained by averaging 3 times with random initialization, where the accuracy and macro-averaged F1 were adopted as the evaluation metrics.

4.1.3. Baseline

To comprehensively evaluate the performance of the proposed AIEN, we compared our model with a range of baselines and state-of-the-art models, as listed below:

• SVM [35] is a traditional method based on complicated feature engineering.
• LSTM [36] obtains the extracted features through LSTM and then uses the obtained softmax function to obtain the emotion polarity.
• The MemNet [37] uses a multi-hop architecture and handles contexts as external memory.
• AOA [38] migrates the attention-over-attention method from the field of machine translation and applies it to the field of ABSA.
• The IAN [20] proposes an interactive attention network model based on LSTM and the attention mechanism, modeling target and context, respectively, then interacting with them, and finally, obtaining the results.
• The TNet-LF [39] proposes a technique for producing target-specific word representations that incorporates a mechanism for keeping the original contextual data from the RNN layer.
• The AEN [14] eliminates recurrence and models the relationship between the contexts and the specific targets using an attentional encoder network.
• The ASGCN [12] proposes a graph convolutional network (GCN) on the dependency tree of sentences to make use of syntactic information and word dependence.
• The AEGCN [13] proposes the multi-head attention and an improved graph convolutional network built over the dependency tree of a sentence.
• The SK-GCN [40] utilizes both the syntax-based and knowledge-based GCN to describe dependency trees and knowledge graphs.

In order to fairly evaluate our method, all experiments in this paper extracted the embedding vector with the GloVe pre-trained model. Furthermore, we also used the BERT pre-trained model and fine-tuned it in combination with this task.

4.2. Performance Comparison

The overall performance of all the models is shown in

Table 2. Model comparison results (%). The top two results with each dataset are in bold. We use “N/A” to represent unreported experimental results. The comparison models’ results were retrieved from the original papers.

Model	TWITTER		LAP14		REST14		REST15		REST16
	Accuracy	F1	Accuracy	F1	Accuracy	F1	Accuracy	F1	Accuracy	F1
SVM [35]	63.4	63.3	70.49	N/A	80.16	N/A	N/A	N/A	N/A	N/A
LSTM [36]	69.56	67.7	69.28	63.09	78.13	67.47	77.37	55.17	86.8	63.88
MemNet [37]	71.48	69.9	70.64	65.17	79.61	69.64	77.31	58.28	85.44	65.99
AOA [38]	72.3	70.2	72.62	67.52	79.97	70.42	78.17	57.02	87.5	66.21
IAN [20]	72.5	70.81	72.05	67.38	79.26	70.09	78.54	52.65	84.74	55.21
TNet-LF [39]	72.98	71.43	74.61	70.14	80.42	71.03	78.47	59.47	89.07	70.43
AEN [14]	72.83	69.81	73.51	69.04	80.98	72.14	N/A	N/A	N/A	N/A
ASGCN-DT [12]	71.53	69.68	74.14	69.24	80.86	72.19	79.34	60.78	88.69	66.64
ASGCN-DG [12]	72.15	70.4	75.55	71.05	80.77	72.02	79.89	61.89	88.99	67.48
AEGCN [13]	73.16	71.82	75.91	71.63	81.04	71.32	79.95	60.87	87.39	68.22
SK-GCN [40]	71.97	70.22	73.20	69.18	80.36	70.43	80.12	60.70	85.17	68.08
Our GloVe	74.37	72.78	75.91	71.71	81.90	73.69	80.01	63.70	88.31	69.43
Our BERT	74.86	73.71	78.21	73.39	85.36	78.33	83.58	64.67	90.58	74.49
BERT + FineTune	74.98	73.15	79.01	74.98	86.96	81.82	84.13	71.41	91.40	75.35

, from which several observations can be noted. The AIEN consistently outperformed all compared models on the TWITTER, LAP14, and REST14 datasets.

The AIEN based on the deep learning model had better performance than the traditional machine learning methods. In the above table, the SVM model proposed by Kirishenko uses SVM for classification, which depends on a large amount of manual feature extraction. On the TWITTER, LAP14, and Rest14 datasets, the accuracy of our model was 10.97%, 5.44%, and 1.74% higher than that of SVM without manual feature extraction. This result showed that the deep learning model is applicable to specific aspects of sentiment analysis.

In addition, the mask mechanism can have the effect of preserving context information on specific aspect words, which had some advantages on REST16. Furthermore, we also speculated that the REST16 dataset may have the shortcomings of fuzzy feature information and a large number of noise data, which, to some extent, affected our model. The experimental results also showed that the accuracy of our model on the REST15 dataset was slightly lower than that on SK-GCN, but the AIEN was significantly better than the SK-GCN on the F1 measurement. Compared with the benchmark models, the ASGCN and AEGCN, our model performed better on various datasets, which showed that the idea of using LSTM and the GCN for feature extraction and emotion classification through the multi-head attention mechanism is reasonable.

The pre-trained model based on BERT showed excellent performance, and the final classification accuracy was far higher than all existing ABSA models. At the same time, in order to make the BERT pre-trained model better match our emotion classification task, we fine-tuned the model. By adding a simple linear layer and a softmax output layer, the model accuracy and F1 value on various datasets exceeded the basic pre-trained model, which fully proved the ability of BERT in this task. As an embedded layer, BERT showed further improvement after joining our network, the AIEN. These results proved the effectiveness of the AIEN in capturing important syntactic structures of sentiment analysis.

4.3. Study of our Method

4.3.1. Ablation Experiment

To further understand the impact of each part of our model on the performance, we conducted several ablation experiments, as shown in

Table 3. Ablation study results (%). The best results with each dataset are highlighted in bold.

Model	TWITTER		LAP14		REST14		REST15		REST16
	Accuracy	F1	Accuracy	F1	Accuracy	F1	Accuracy	F1	Accuracy	F1
Our	74.37	72.78	75.91	71.71	81.90	73.69	80.01	63.70	88.31	69.43
w/o mask	73.98	72.61	73.09	68.29	79.82	69.51	78.59	60.93	87.53	67.77
w/o GCN	73.69	72.20	75.28	70.95	81.69	73.53	79.58	62.24	88.04	68.07
w/o context	72.10	70.17	73.14	68.41	80.53	71.09	79.52	56.87	87.71	66.19
w/o MHIA	72.88	71.69	73.77	69.37	80.50	70.00	80.50	62.77	88.04	65.73

. All the data in the w/o context experiments were inferior to the benchmark because the context contains aspect lexical structure information and semantic correlation, which had an impact on the performance of the model. In addition, the result of w/o the GCN experiments also proved that the GCN method can be harmoniously combined with our proposed model and plays a positive role. The experimental results of w/o mask further emphasized the importance of the mask layer and proved that the target-oriented feature extraction plays a key role in the ABSA task.

From the result without MHIA, the performance of the Rest15 dataset was improved after deleting MHIA. The possible reason is that, compared with the other datasets, the Rest15 dataset is more sensitive to syntactic information. Using MHIA will cause a small part of the Rest15 dataset to lose key feature information.

4.3.2. Parameter Experiment

Since the design of the AIEN model involves MHA and the GCN, we investigated the impact of the number of headers of MHA and the number of layers of the GCN on the model performance and checked the accuracy and macro-F1 of the AIEN on the LAP14 dataset. The results are shown in Figure 3 and Figure 4. When the attention headers reached three and the number of GCN layers was two, the model obtained the best performance. This proved the rationality of selecting parameters in the design of the AIEN model.

4.3.3. Case Study

In this section, to better understand how the AIEN works, we further show the advantages of our proposals through some typical examples. We list some examples from the test set that have typical characteristics in Figure 5. In Sentence (a), “I have not a bad thing to say about this place.”, there is a negative word: not. This can easily lead to the model making the wrong predictions. Our model can obtain the correct polarity, unaffected by negation words that do not represent negation here. In Sentence (b), “great food but the service was dreadful!”, there are two different aspects of food and service, which brings additional difficulty to detecting the target semantics. Our model can distinguish different aspects of different emotion polarities. In the last instance (c), “I’m not necessarily fanatical about this place, but it was a fun time for low prices”, the sentence structures are long and complex, and it may be difficult for existing models to make correct judgments to correctly predict the polarity. However, our model can process these sentences well with the help of modules such as the attention mechanism and the GCN to make the correct sentiment polarity judgment.

5. Conclusions

Due to the lack of research on syntactic features and high-level interaction features simultaneously in traditional works, this paper proposed the AIEN model to overcome the problem. Through LSTM and the GCN layer, we obtained multi-dimensional features including the semantic features of the aspect itself, the context-aware semantic features, and the syntactic features. Based on these features, we constructed the multi-dimensional representation of a specific aspect, which was composed of the new high-level interactive features. These generated multi-dimensional features were used for the final prediction of the ABSA task. We conducted extensive experiments on five datasets and compared the experimental performance with other advanced methods. The experimental results showed that our proposed approach can significantly improve the effectiveness of the model. In future work, we will extend the method proposed in this paper to study the performance of multiple aspects to verify the scalability of this method.