Graph Convolutional Networks with POS Gate for Aspect-Based Sentiment Analysis

Abstract

We make daily comments on online platforms (e.g., social networks), and such natural language texts often contain sentiment (e.g., positive and negative) for certain aspects (e.g., food and service). If we can automatically extract the aspect-based sentiment from the texts, then it will help many services or products to overcome their limitations of particular aspects. There have been studies of aspect sentiment classification (ASC) that finds sentiment towards particular aspects. Recent studies mostly adopt deep-learning models or graph neural networks as these techniques are capable of capturing linguistic patterns that contributed to performance improvement in various natural language processing tasks. In this paper, for the ASC task, we propose a new hybrid architecture of graph convolutional network (GCN) and recurrent neural network. We design a gate mechanism that jointly models word embeddings and syntactic representation of sentences. By experimental results on five datasets, we show that the proposed model outperforms other recent models and also verify that the gate mechanism contributes to the performance improvement. The overall F1 scores that we achieved is 66.64∼76.80%.

1. Introduction

People often write comments on social network services (SNS) or websites of arbitrary services (e.g., online shopping mall), and it is obvious that it will be significantly helpful for service providers if it is possible to get the opinions or customer satisfaction level from several customer comments. Such comments are mostly written in natural language texts, and an example is shown in Figure 1; there are two aspects (e.g., food and service) with different sentiment (e.g., positive and negative). There have been studies for analyzing the natural language texts containing sentiment and aspects, and such a research field is known as aspect-based sentiment analysis (ABSA). The ABSA contains few tasks [1]: (1) entity-based aspect identification, (2) extraction of linguistic expressions that refer to arbitrary aspects, and (3) aspect sentiment classification (ASC). We aim at the task of ASC that extracts aspect-based sentiment from sentences. A sentence may contain multiple aspects, each of which corresponds to different sentiment, and the aspect-based sentiment (e.g., ‘service’: negative, and ‘food’: positive) will help to examine customer satisfaction; for example, the restaurant owner will investigate what is wrong with the service in the restaurant. Because previous studies might be interpreted in the wrong way if we use different definitions of sentiment, we here follow the definition of Giuseppe D’Aniello et al. [2]; sentiment is a durable emotional disposition that is developed by the user with respect to an aspect.

For the task of ASC whose goal is to identify aspects and sentiment expressed towards each aspect, there have been studies using machine learning models. Kiritchenko et al. [3] used support vector machine (SVM) [4] and achieved 80% accuracy for a dataset of SemEval 2014 Task 4 subtask 2 restaurant domain (REST14). Mubarok et al. [5] used naive Bayes classifier with a feature selection algorithm, and they achieved a 78% of F1 score for the REST14 dataset. Although these studies have shown successful performance, they are limited because they require significant effort in feature engineering.

Deep learning (DL) models are favorable as they find arbitrary features automatically from data [6]. There are a few well-known types of DL models: multi-layered perceptron (MLP) [7], recurrent neural networks (RNN) [8], convolutional neural networks (CNN) [9], and attention mechanism [10]. These different DL models have their own merits, as discussed in [11]; for example, some studies employed CNN to tackle the task of sentiment classification as the CNN is known to be efficient and effective in analyzing local patterns [12,13]. In particular, attention-based RNN architecture was often adopted in many studies of the ASC task as the attention-based architecture has shown its effectiveness in analyzing sequential patterns beneath the word sequences of natural language texts [10,14]. Wang et al. [15] proposed an attention-based RNN model using long short-term memory (LSTM) [16], and achieved 78% accuracy for REST14. Fan et al. [17] proposed bi-directional LSTM (Bi-LSTM) with a fine-grained attention mechanism, that captures the word-level interaction between aspect and context. They achieved 81% accuracy for REST14.

Recently, a graph neural network is widely used to incorporate graph-like patterns such as dependency between words for the ASC task. Xing and Tsang [18] proposed a hybrid architecture of LSTM and a graph neural network, and adopted other resources (e.g., DBpedia and Wikipedia) to train their model. They achieved 87% accuracy for REST14. Huang et al. [19] utilized pre-trained Bidirectional Encoder Representations from Transformers (BERT) [20] and graph attention network (GAT) on the dependency tree, and achieved 85% accuracy for REST14. Using context-aware pre-trained language models (e.g., BERT) has drawn much attention as it allows us to achieve better accuracy; however, it requires relatively large computational cost compared to context-free language models (e.g., Word2Vec [21] and Glove [22]), as reported in [23]. There are studies that adopt neither context-aware pre-trained models nor other resources. Zhang et al. [24] proposed building a graph convolutional network (GCN) over the dependency tree of a sentence to exploit syntactical information and word dependencies, and achieved 80% accuracy for the REST14 dataset. Xiao et al. [25] proposed a model of multi-head attention and an attentional-encoding-based GCN of a dependency tree, and achieved 81% accuracy for REST14. Bai et al. [26] proposed a relational graph attention network (RGAT) to incorporate typed syntactic dependencies. They also utilized part-of-speech (POS) tags that are known to convey rich syntactic patterns, and achieved 83% accuracy for REST14. However, they just used embeddings of POS tags, and did not jointly model the POS embeddings and word embeddings.

Word embeddings have often been jointly modeled with syntactic representation (e.g., POS tags) [27,28] because it allows us to capture more sophisticated patterns between semantic and syntactic representation. For example, the word ‘good’ has a different meaning for different POS tags (e.g., adverb or noun), and it has multiple meaning even for the same POS tag. In this paper, we assume that modeling semantic and syntactic representations jointly will contribute to performance improvement for the ASC task.

We propose a multi-layered GCN-BiLSTM architecture with a gate mechanism that jointly incorporates POS representations and word representations, and proves the effectiveness of the model by experimental results. The proposed architecture consists of GCN and BiLSTM, where the BiLSTM extracts sequential patterns in the given texts and the GCN analyzes syntactic information. The gate mechanism incorporates the syntactic information and POS tags together, which makes the entire model work more accurately.

2. Method

Figure 2 depicts the framework of our proposed model, namely POS-gated graph convolutional networks (PGGCN), that is composed of two embedding layers (e.g., word embedding and POS embedding) and a hybrid of GCN-BiLSTM layers with the attention mechanism. The word embedding layer and the POS embedding layer convert the given input sequence of words (or POS tags) into real-valued vectors in a different space; thus, we will get two embedding vectors from the embedding layers. Each of the two embedding vectors is delivered to BiLSTM that captures sequential patterns in the embedding vector. The multi-layered GCN and POS-gated attention mechanism exploit the sequential patterns extracted from the BiLSTM models, and generates final representation. Details of each part of the proposed model are provided in the following subsections.

2.1. Embeddings and Bi-LSTM

Given a sentence $\{w_1,w_2,...,w_n\}$ where $w_i$ indicates i-th word, we get a sequence of POS tags $\{p_1,p_2,...,p_n\}$ using NLTK [29] POS tagger. Aspect is a sub-sequence of the word sequence, and there may exist one or more aspects in a sentence. The words and POS tags are passed to the word embedding layer and POS embedding layer. The embedding layers embed the words and POS tags in vector spaces with word embedding matrix and POS embedding matrix $E^w\in {\mathbb{R}}^{d_e\times \left|V\right|},E^p\in {\mathbb{R}}^{d_e\times \left|P\right|}$, where $d_e$ is the dimension of embedding, $\left|V\right|$ denotes the vocabulary size, and $\left|P\right|$ indicates the number of unique POS tags. Pre-trained GloVe [22] is used for the word embedding, and the POS embedding is generated using training data with POS tags. The word embeddings and POS embeddings are passed to two different bi-directional LSTM layers (i.e., Word LSTM and POS LSTM), and they yield a sequence of word hidden state vectors $\{h_1^w,h_2^w,...,h_n^w\}$ and a sequence of POS hidden state vectors $\{h_1^p,h_2^p,...,h_n^p\}$, where $h_t^{·}\in {\mathbb{R}}^k$ is a concatenation of hidden states of forward and backward LSTM at t-th word.

2.2. Gcn and Aspect-Specific Masking

We created an adjacency matrix A between words based on a dependency tree obtained by SpaCy toolkit [30]. The matrix A and word hidden states are fed into the L-layered GCN as shown in the left side of Figure 2. For every l-th layer, i-th word representation $h_i^l$ is updated by incorporating representations of its adjacent words with normalization factor [31] as follows:

$${\tilde{h}}_i^l=\sum _{j=1}^n{A}_{ij}{W}^l{g}_j^{l-1}$$

(1)

$$h_i^l=Relu({\tilde{h}}_i^l/(d_i+1)+b^l)$$

(2)

where $g_j^{l-1}$ is j-th word representation of previous layer, $d_i={\sum }_{j=1}^n{A}_{ij}$ is a degree of the i-th word in the dependency tree. Note that $h_i^l$ is computed using $g_j^{l-1}$ that is a position-aware word representation [32,33,34] as follows.

$$g_i^{l-1}=p_i{h}_i^{l-1}$$

(3)

$$p_i=\left\{\begin{array}{cc}1-\frac{r+1-i}{n}& 1\le i<r+1\\ 0& r+1\le i\le r+m\\ 1-\frac{i-r-m}{n}& r+m<i\le n\end{array}\right.$$

(4)

where $r+1$ is the starting position of an aspect, m is a length of the aspect, and $p_i\in \mathbb{R}$ is the position-aware weight of the i-th word.

At the last GCN layer, we get the final hidden state vectors $H^L=\left\{h_i^L\right\}$ where $1\le i\le n$. As shown in the top-left corner in Figure 2, we perform aspect-specific masking that removes non-aspect hidden state vectors; the outputs will be $H_{mask}=\{0,...,h_{r+1}^L,...,h_{r+m}^L,...,0\}$.

2.3. Part-of-Speech Gate

POS gate is designed to regularize the POS hidden states based on word hidden states. For the i-th word, it takes $h_i^w$ and $h_i^p$ as input, and generates $h_i^{pos}\in {\mathbb{R}}^m$ as follows.

$$h_i^g=W^g·tanh(h_i^w+h_i^p)$$

(5)

where $W^g$ is a trainable matrix of ${\mathbb{R}}^{m\times k}$. The gate output is used together with GCN output to generate attention scores as shown in top-right corner of Figure 2.

2.4. Attention-Based Prediction

We compute attention scores for t-th position by incorporating $H_{mask}$ and $H_{gate}=\left\{h_i^g\right\}$ through a retrieval-based attention mechanism [24].

$${\beta }_t=\sum _{i=1}^n{h}_t^{g\top }{h}_i^L=\sum _{i=r+1}^{r+m}{h}_t^{g\top }{h}_i^L$$

(6)

$${\alpha }_t=\frac{exp\left({\beta }_t\right)}{{\sum }_{i=1}^{n}exp\left({\beta }_i\right)}$$

(7)

Finally, sentiment $\widehat{y}$ is predicted based on $h_{att}$ as formulated below.

$$h_{att}=\sum _{i=1}^n{\alpha }_i{h}_i^g$$

(8)

$$\widehat{y}=softmax(W{h}_{att}+b)$$

(9)

2.5. Training

We train the model using the cost function C with $L_2$-regularization [35] and cross-entropy:

$$C=-\sum _{i=1}^N\sum _{j=1}^S{y}_i^j·log\left({\widehat{y}}_i^j\right)+\gamma {∥\Theta ∥}_2$$

(10)

where N is the number of instances, S is the number of classes, ${\widehat{y}}_i^j$ means the output of the model, $y_i^j$ is the ground truth, and $\gamma$ denotes the $L_2$-regularization coefficient.

3. Experiments

3.1. Datasets and Experimental Settings

We conduct experiments on five datasets: restaurant reviews of SemEval 2014, 2015, and 2016 (i.e., REST14, REST15, and REST16) [1,36,37], laptap review of SemEval 2014 (LAPTOP), and Twitter review data (TWITTER) [38]. The Statistics are summarized in

Table 1. Statistics of the datasets, where $m/n$ indicates the number of training instances m and the number of test instances n.

Dataset	Positive	Neutral	Negative
REST14	2164/728	637/196	807/196
REST15	1178/439	50/35	382/328
REST16	1620/597	88/38	709/190
LAPTOP	994/341	464/169	870/128
TWITTER	1561/173	3127/346	1560/173

.

The embedding dimension is set to 300, and we borrowed pre-trained GloVe vectors [22] to initialize the word embeddings. The number of GCN layers $L=2$. We utilized Adam optimizer with the initial learning rate of ${10}^{-3}$. The $L_2$ regularization coefficient $\gamma$ is ${10}^{-5}$, and mini-batch size is 32. We use accuracy and macro-averaged F1 score as evaluation metrics, and all results are obtained by averaging 20 independent runs; the F1 score and accuracy are, of course, not the perfect measure, and alternative ways of F-measure [39] may be considered when they are more studied in the future. Our model is compared with others [18,19] that do not employ pre-trained context-aware language models (e.g., BERT) or any other resources (e.g., dataset or knowledge-base).

3.2. Results

Table 2. Averaged accuracies and F1 scores with five datasets (e.g., REST14, REST15, REST16, LAPTOP, and TWITTER), where results with ♮ are borrowed from [24]. The best results for each dataset are in bold. The GCN-based models generally gave better performance than the others, and our proposed PGGCN achieved the best performance.

MODEL	REST14		REST15		REST16		LAPTOP		TWITTER
	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1
TD-LSTM [41]	78.00	68.43	76.39	58.70	82.16	54.21	71.80	68.46	69.89	66.21
ATAE-LSTM [15]	78.60	67.02	78.48	62.84	83.77	61.71	68.88	63.93	70.14	66.03
MemNet ${}^{♮}$ [33]	79.61	69.64	77.31	58.28	85.44	65.99	70.64	65.17	71.48	69.90
AOA ${}^{♮}$ [42]	79.97	70.42	78.17	57.02	87.50	66.21	72.62	67.52	72.30	70.20
IAN ${}^{♮}$ [43]	79.26	70.09	78.54	52.65	84.74	55.21	72.05	67.38	72.50	70.81
TNet-LF ${}^{♮}$ [32]	80.42	71.03	78.47	59.47	89.07	70.43	74.61	70.14	72.98	71.43
ASGCN ${}^{♮}$ [24]	80.77	72.02	79.89	61.89	88.99	67.48	75.55	71.05	72.15	70.40
AEGCN [25]	81.04	71.32	79.95	60.87	87.39	68.22	75.91	71.63	73.16	71.82
MGGCN [44]	81.16	71.73	80.19	64.62	88.96	69.48	75.80	71.75	73.41	71.89
PGGCN	83.84	76.80	82.47	66.64	90.42	74.49	77.74	74.56	74.57	72.01

shows the overall performance of our model PGGCN and other various models. The PGGCN outperformed other models with all datasets. There are a few interesting points. First, all GCN-based models (e.g., ASGCN, AEGCN, and MGGCN) showed better performance than others; this might imply that the GCN has better ability to grasp underlying syntactic patterns, so it contributes to performance improvement. Second, with the TWITTER dataset, the performance gap between PGGCN and others is smaller compared to other datasets of SemEval. The TWITTER dataset has quite different nature from others; it is collected from social network platform and has much more ‘neutral’ instances, as shown in Table 1. Such many ‘neutral’ instances confuse the models, and may drive the performance gap to be smaller. This is reasonable as ‘neutral’ might not even exist as we are always feeling something [40]. We also observed that all models generally gave better results with greater datasets, except for the TWITTER dataset; this might be related to the many ‘neutral’ instances of the TWITTER dataset.

We performed experiments for the PGGCN without position weight, aspect-specific masking, GCN layers, and POS gate, to check its impact on the performance. The result is summarized in

Table 3. Averaged accuracies and F1 scores of PGGCN without some components, where ‘w/o POS gate’, ‘w/o position’, ‘w/o mask’, and ‘w/o GCN’ indicate PGGCN without POS gate, without position weights, without aspect-specific masking, and without GCN layers, respectively. The PGGCN without any component gave the worst results, which implies that all components including the POS gate are important.

MODEL	REST14		REST15		REST16		LAPTOP		TWITTER
	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1
w/o POS gate	83.39	76.37	80.81	65.14	89.45	73.79	75.86	72.26	73.12	71.62
w/o position	82.68	74.82	80.81	64.30	89.45	71.11	75.39	71.07	73.84	72.29
w/o mask	79.29	69.85	78.04	63.52	87.50	64.99	72.41	67.88	72.40	71.13
w/o GCN	81.25	72.47	81.00	63.34	87.82	69.67	74.61	70.69	72.40	71.30

; it shows that they contributed to performance improvement. In particular, it proves the usefulness of the POS gate, which is the contribution of this paper; modeling word and POS representations jointly helps to get better results on the ASC task. We also conducted experiments with a varied number of GCN layers, and its accuracies and F1 scores are shown in Figure 3 and Figure 4, respectively. We found that five or more GCN layers did not give any performance gain, and the best number of layers was 2.

4. Conclusions

As a solution for the ASC task, we propose a hybrid architecture of GCN-BiLSTM, namely PGGCN, with a gate mechanism that jointly exploits word representations and POS representations. We compared PGGCN with other recent models by experiments with five datasets, and showed that PGGCN outperformed the others. Amongst the many models, we found that GCN-based models outperformed others, which indicates that syntactic patterns discovered by GCN has an impact on the performance. We also observed that the models generally gave better results with greater datasets, but the models gave poor results when the dataset has many ‘neutral’ instances. The experimental results of PGGCN without the POS gate showed the effectiveness of the POS gate. We did not utilize any pre-trained language models [20,45] that have shown promising results recently, so we will investigate a way of combining our model with pre-trained context-aware language models as future work.