Modeling the Paraphrase Detection Task over a Heterogeneous Graph Network with Data Augmentation ^†

Abstract

Paraphrase detection is a Natural-Language Processing (NLP) task that aims at automatically identifying whether two sentences convey the same meaning (even with different words). For the Portuguese language, most of the works model this task as a machine-learning solution, extracting features and training a classifier. In this paper, following a different line, we explore a graph structure representation and model the paraphrase identification task over a heterogeneous network. We also adopt a back-translation strategy for data augmentation to balance the dataset we use. Our approach, although simple, outperforms the best results reported for the paraphrase detection task in Portuguese, showing that graph structures may capture better the semantic relatedness among sentences.

1. Introduction

Paraphrase detection is a Natural-Language Processing (NLP) task that aims to automatically identify whether two sentences convey the same meaning. Bhagat and Hovy [1] define paraphrase as sentences or phrases that convey the same meaning using different wording. Moreover, these sentences represent alternative surface forms in the same language, expressing the same semantic content of the original forms [2].

Formally, a paraphrase may be modeled as a mutual (or bidirectional) entailment between a text T and a hypothesis H in the form, $T\to H$ and $H\to T$, that means T entails H and H entails T. For example, given a text T and a hypothesis H below, one may claim that they are paraphrases of each other, since $T\to H$ and $H\to T$.

T 

It is a strange term, but we have become used to it.

H 

This term is strange; however, we are accustomed to it.

According to Anchiêta and Pardo [3], studies that have focused on the paraphrase detection task for the Portuguese language (as defined above) are rare. A reason for this is the lack of large available corpora of paraphrases. As a consequence, there are few developed methods for paraphrase identification that could be useful for varied NLP tasks that might benefit from such knowledge, as semantic parsing [4], machine translation [5], automatic summarization [6], question answering [7] and plagiarism detection [8], among others.

To try to overcome such barriers, researchers developed and used the ASSIN corpus [9], which is focused on the textual entailment recognition task, but includes paraphrase examples. Formally, entailment recognition is the task of deciding whether the meaning of one text may be inferred from another [9].

The existing works that aim to detect paraphrase sentences in Portuguese [3,10], model this task as a machine-learning solution, building feature-value tables and training and testing classifiers. For a new sentence pair, features are computed and fed into the classifier to predict if the two sentences are paraphrases of each other. These approaches may suffer from two drawbacks. In the first one, the features may not capture well the semantics of the sentence pairs, producing unsatisfactory results. In the second, the authors apply sampling techniques to mitigate the unbalance issues of the ASSIN corpus, aiming to get more balanced data to improve the results of their models. These under- or over-sampling techniques may suffer from some shortcomings. In the over-sampling, the minority class can lead to model overfitting, introducing duplicate instances from a pool of instances that is already small [11]. On the other hand, in the under-sampling, the majority class can end up leaving out important instances that provide important differences between the two classes [12]. Other strategies that make use of synthetic data also suffer from criticism on the quality of the generated data.

To fulfill these gaps and explore other approaches for paraphrase detection, in this paper, inspired by Sousa et al. [13], we model the paraphrase detection task as a heterogeneous network. In this network, nodes represent tokens and sentence pairs, and the edges link the two node types. Networks/graphs have shown to be a powerful data structure that may capture well the relationship among the objects of interest [14].

Based on the network, we feed and train a classifier to predict if two sentences are paraphrases of each other. To evaluate our method, we use the ASSIN corpus. However, instead of applying a sampling technique to balance it, we adopt a back-translation strategy [15] for data augmentation to balance the data. This strategy maintains the original sentence pairs from the ASSIN corpus and add real sentences from another corpus with good translation quality. Our proposed method outperforms the best reported results, both in F-score and accuracy measures. Furthermore, the back-translation strategy helps to produce better models.

The remaining of this paper is organized as follows. Section 2 briefly presents the related work. In Section 3, we show the used corpora. Section 4 details our methodology to balance the ASSIN corpus and to model the problem. Section 5 presents the conducted experiments and obtained results. Finally, in Section 6, we conclude the paper, giving directions for future work.

2. Related Work

As pointed by Anchiêta and Pardo [3], few approaches strictly tackle the paraphrase detection task for the Portuguese language. Most of the research is on entailment identification that, according to Souza and Sanches [10], is different from paraphrase detection. Thus, following Souza and Sanches [10], we focus on the paraphrase detection task.

Consoli et al. [16] analyzed the capabilities of the coreference resolution tool CORP [17] for identification of paraphrases. The authors used CORP to identify noun phrases that may help to detect paraphrases between sentence pairs. They evaluated their method on 116 sentence pairs from the ASSIN corpus, achieving 0.53 F-score.

Rocha and Cardoso [18] modeled the task as a supervised machine-learning problem. However, they handled the issue as a multi-class task, classifying sentence pairs into entailment, none, or paraphrase. Thus, they employed a set of features of the lexical, syntactic, and semantic levels to represent the sentences in numerical values, and fed these features into some machine-learning algorithms. They evaluated their method on the training set of the ASSIN corpus, using both European and Portuguese partitions. The method obtained 0.52 of F-score using an SVM classifier.

Souza and Sanches [10] also dealt with the problem using a supervised machine-learning strategy. However, their objective was to explore sentence embeddings for this task. They used a pre-trained FastText model [19] and the following features: the average of the vectors, the value of Smooth Inverse Frequency (SIF) [20], and weighted aggregation based on Inverse Document Frequency (IDF). With these features, their method reached 0.33 of F-score using an SVM classifier on balanced data of the ASSIN corpus for European and Portuguese partitions.

Cordeiro et al. [21] developed a metric named Sumo-Metric for semantic relatedness between two sentences based on the overlapping of lexical units. Although the authors evaluated their metric on a corpus for the English language, the metric is language-independent.

Anchiêta and Pardo [3] explored the potentialities of four semantic features to identify paraphrase sentences. They computed the similarity of two encoded sentences as a graph using a semantic parser [22] and a semantic metric [23], the value of Smooth Inverse Frequency (SIF) [20], the cosine distance between two embedded sentences, and the value of the Word Mover’s Distance (WMD) [24] between two embedded sentences. From these features, they trained an SVM classifier and obtained 0.80 F-score on the balanced ASSIN corpus.

For the English language, according to Mohamed and Oussalah [25], most of the research is categorized into three high levels, namely: corpus-based, knowledge-based, and hybrid methods. Here, in order to have a panoramic view of the achieved contributions for this language and to allow (indirect) comparisons to Portuguese state of the art, we briefly present the best results in the literature.

Mohamed and Oussalah [25] adopted a hybrid method addressing the problem of evaluating sentence-to-sentence semantic similarity when the sentences contain a set of named entities. The authors aimed to distinguish the computation of the semantic similarity of named entity tokens from the rest of the sentence text based on the integration of word semantic similarity derived from WordNet taxonomic relations [26], and named entity semantic relatedness inferred from Wikipedia entity co-occurrences and underpinned by Normalized Google Distance [27]. This approach reached 85.2% F-score on the MSRP corpus [28].

Ji and Eisenstein [29] adopted a corpus-based approach and used a distributional similarity model by designing a discriminative term-weight metric called TF-KLD. This metric outperforms the TF-IDF weighting scheme by re-weighting the sentence-term matrix in a different way. The authors evaluated their method on the MSRP corpus and achieved an 85.96% F-score.

Issa et al. [30] also adopted a corpus-based approach. For that, they combined Latent Semantic Analysis (LSA) [31] with Abstract Meaning Representation (AMR) [32] parsing. In this combination, the authors re-weighted the LSA sentence-term matrix according to a probability distribution over the AMR nodes, which was accomplish by means of the PageRank algorithm [33]. With this strategy, the authors reached 90% F-score on the MSRP corpus.

As we can see, our graph-based method is different from the literature both in the Portuguese and English languages. Moreover, our approach does not make use of external knowledge for computing sophisticated features: it uses only the sentences and tokens of the corpus.

3. Corpora

3.1. The ASSIN Corpus

ASSIN (Semantic Similarity Assessment and textual INference) [9] is a corpus with semantic similarity score and entailment annotations for sentence pairs in Portuguese, which was released for a shared task of the PROPOR 2016 Conference (http://nilc.icmc.usp.br/assin/). The corpus has 10,000 sentence pairs, half of which is written in Brazilian Portuguese and half in European Portuguese. Moreover, each language has 2500, 500, and 2000 pairs for training, development, and testing, respectively, as presented in

Table 1. The ASSIN corpus.

Portuguese Language	Training	Development	Testing
Brazilian	2500	500	2000
European	2500	500	2000

.

Besides that organization, the sentence pairs of the corpus are labeled with inference classification, as entailment, none, and paraphrase, as shown in

Table 2. Frequency of the labels in the ASSIN corpus.

Label	Brazilian Port.			European Port.			Total #	Proportion %
	Train.	Dev.	Test.	Train.	Dev.	Test.
Entailment	437	92	341	613	116	481	2080	20.80
None	1947	384	1553	1708	338	1386	7316	73.16
Paraphrase	116	24	106	179	46	133	604	6.04

. One can see that the corpus is unbalanced concerning the paraphrase label, since the proportion of entailment and none labels is much higher than the paraphrase label.

3.2. The MSRP Corpus

MSRP (Microsoft Research Paraphrase) [28] is a widely used corpus for the paraphrase detection task for English. The sentence pairs are annotated in a binary strategy, i.e., as paraphrases or non-paraphrases. The corpus has 5801 sentence pairs, where 3900 are paraphrases and 1901 are non-paraphrases, as exposed in

Table 3. Splits of the MSRP Corpus.

Label	Training	Testing	Total #	Proportion %
Paraphrase	2753	1147	3900	67.54
Non-paraphrase	1323	578	1901	32.46

. As we can see, the MSRP corpus is unbalanced concerning the non-paraphrase label.

In what follows, we detailed the developed methods for paraphrase identification and our strategy to mitigate the unbalance of the ASSIN corpus.

4. A New Method for Paraphrase Identification

4.1. Balancing the ASSIN Corpus

To formulate the paraphrase identification task as a binary classification problem, we did a modification to the ASSIN corpus, since it has three labels. Thus, we joined the entailment and none labels of the corpus into one unique label named “non-paraphrase”, which is our negative class, as presented in

Table 4. New configuration of the ASSIN corpus.

Label	Training	Development	Testing	Total #	Proportion %
Non-paraphrase	4705	930	3761	9396	93.96
Paraphrase	295	70	239	604	6.04

.

As we can see, the new configuration of the corpus is unbalanced concerning non-paraphrase label. Aiming to balance the corpus, we used the MSRP corpus. For that, we adopted a back-translation strategy [15] to translate the sentences of the MSRP corpus from English to Portuguese, as illustrated in Figure 1.

According to this figure, we translated the original sentences from the MSRP corpus to Portuguese, using the machine translation model provided by the Google Translate API (https://cloud.google.com/translate/). We adopted this model due to the good results achieved by other researchers [34,35] when using it. Next, we translate the Portuguese sentences back to English, because there are no Portuguese reference sentences to evaluate the quality of the translations. This way, we may measure the quality of the translations, comparing the original sentences (reference sentences) with the back-translated sentences (hypothesis sentences). To compute the quality of the translations, we calculated the harmonic mean between ROUGE (We used the F-score of the ROUGE-L.) [36] and BLEU [37] metrics, as in Equation (1).

$$F1=2\times \frac{rouge\times bleu}{rouge+bleu}$$

(1)

We achieved 0.844 mean value (with a 0.08 standard deviation) when comparing the reference sentences to the hypothesis sentences. Taking into account the state-of-the-art results in machine translation [38], which achieves 35 points on the BLEU score, we may consider that the translated sentences from English to Portuguese have a good quality due to reached results both in the mean and the standard deviation. Thus, we used the translated sentences of the MSRP corpus to make the ASSIN corpus less unbalanced.

Table 5. ASSIN corpus plus the translated sentences of the MSRP corpus.

Label	Training	Development	Testing	Total #	Proportion %
Non-paraphrase	4705	930	3761	9396	67.60
Paraphrase	2295	823	1386	4504	32.40

presents the ASSIN corpus plus the translated sentences of the MSRP corpus.

To create this less unbalanced corpus, we got the 2753 paraphrastic sentences from the training set of the MSRP corpus and put 2000 into the training set and 753 into the development set of the ASSIN corpus, respectively. As a consequence of these operations, the training set will have 2295 sentences and the development set will have 823 sentences. Moreover, we also got the 1147 paraphrastic sentences from the testing set of the MSRP corpus and put them into the testing set of the ASSIN corpus. As a result of that operation, the testing set will have 1386 sentences. After these procedures, we obtained a corpus in Portuguese with similar proportions in relation to the MSRP corpus.

4.2. Modeling the Paraphrase Identification Task

To extract features, we first model the paraphrase identification task over a heterogeneous graph network. This network contains abundant information with structural relations (edges) among multi-typed nodes as well as unstructured contend associated with each node [39,40]. It has been used to automate feature engineering tasks, facilitating machine-learning tasks, and proving good results for other tasks as helpfulness prediction, text classification and scientific impact measurement, among others [13,41,42,43].

We borrowed the formulation of a heterogeneous network from Chang et al. [42] and adapted it to our purpose. Our network may be viewed as a graph $G=(V,E)$, where $V=\{v_1,...,v_n\}$ is a set of vertices and E is a set of edges, where an edge $e_{i,j}\in \{1,...,n\}$ belongs to the set E if and only if an undirected and unweighted link exists between nodes i and j. Moreover, the graph G is associated with an object type mapping function $f_v:V\to O$, where O represents object sets and each node $v_i\in V$ belongs to one particular object type as $f_v\left(v_i\right)\in O$.

In our network, we defined two node types and two constraints. In the former, the nodes are tokens and sentence pairs, while in the latter the constraints are: (i) there is no link among token nodes and (ii) there is no link among sentence pair nodes. Thus, we link only token nodes with sentence pair nodes. We present the general scheme and an overview of the network in Figure 2 and Figure 3, respectively.

As one can see, the edges are undirected and unweighted, and a sentence pair node may share several token nodes whenever the token is in the sentence pair, i.e., the edges between token nodes and sentence pair nodes are based on word occurrence in sentence pairs.

To extract the features regarding the network object classes, we applied a regularization method. Regularization is a kind of transductive classification method that aims to find a set of labels, minimizing a cost function and satisfying two conditions: (i) the method needs to be consistent with the set of labels manually annotated and (ii) the method needs to be consistent with the network topology, considering that nearest neighbors tend to have the same labels [14].

We tested three regularization methods: Gaussian Fields and Harmonic Function (GFHF) [44], Learning with Local and Global Consistence (LLGC) [45], and GnetMine [14]. The regularization algorithms require some nodes to be pre-labeled with their specific classes. Thus, one of the differences between these methods is if they modify these pre-labeled nodes. For example, the GFHF method does not modify the pre-labeled nodes, whereas the LLGC and GnetMine methods do. Besides, the GnetMine method works only on the heterogeneous networks, while the GFHF and LLCG methods work both on heterogeneous and homogeneous networks.

As a result, the regularization methods produce values related to coordinates for each object in the network, and these values may be used to feed a supervised machine-learning algorithm to learn and predict labels [46].

Table 6. Result of a regularization method.

Id	Value 1	Value 2	Label
0	0.002959	0.005986	1
29	0.002750	0.005323	1
54	0.003350	0.006426	0
69	0.003953	0.006793	0

presents an example of the output of a regularization method, where id is the identifier of the object, values refer to coordinates of each object in the network, and label 1 is a paraphrase, while label 0 is a non-paraphrase.

4.3. Formulating the Paraphrase Identification Task

As before mentioned, we formulated the paraphrase identification task as a binary classification problem. For that, to train machine-learning algorithms, our method receives as input data the extracted features from a regularizer method in the form of $(x_1^{\left(i\right)},x_2^{\left(i\right)},b^{\left(i\right)})$ for $i\in \left[n\right]$, where n is the number of training sentences, $x_1^{\left(i\right)}$ and $x_2^{\left(i\right)}$ are the input sentences, and $b^{\left(i\right)}\in \{0,1\}$ is the label, referring to a binary classification that informs whether $x_1^{\left(i\right)}$ and $x_2^{\left(i\right)}$ are paraphrases of each other. In summary, the aim is to learn a classifier c that, given unseen sentence pairs, i.e., a set of sentences S, classifies whether they are paraphrases, as in Equation (2).

$$c:S\times S\to 0,1$$

(2)

We tested four machine-learning algorithms, namely: Support Vector Machine (SVM), Naïve Bayes (NB), Decision Tree (DT), and Neural Network (NN). In what follows, we detail our experiments and the obtained results.

5. Experiments and Results

To evaluate our approach, we used the balanced ASSIN corpus with the translated sentences of the MSRP corpus, as depicted in Table 5. Moreover, as we commented before, we tested some classifiers from the Scikit-Learn library [47], as Support Vector Machine (SVM), Naïve Bayes (NB), Decision Trees (DT) and Neural Networks (NN), and we evaluated three regularization methods. Recall that the regularization methods require some nodes to be pre-labeled, so we ranged from 5% to 50% the number of pre-labeled nodes. The regularizers randomly pre-labeled the nodes. Supposing that the percentage of pre-labeled nodes is equal to 5%, it means that 0.25% of each class is randomly pre-labeled.

We achieved the best result with the LLGC regularizer, the NN classifier (We used a Multi-Layer Perceptron (MLP) with 2 hidden layers and 20 neurons in each hidden layer.), and 30% of the pre-labeled nodes on the balanced ASSIN corpus, as depicted in

Table 7. Results of the LLGC regularizer and the NN classifier.

Pre-Labeled (%)	NN
	F-Score		Accuracy
	Paraphrase	Non-Paraphrase	-
5	0.00	0.84	0.73
10	0.00	0.84	0.73
15	0.70	0.90	0.85
20	0.71	0.90	0.85
30	0.74	0.91	0.87
40	0.74	0.91	0.87
50	0.74	0.91	0.87

. It is important to highlight that only the training set is pre-labeled. The regularizer does not have access to labels of the testing set.

As we can see, from the 30% of pre-labeled nodes, both F-score and accuracy remain constants. We believe that the LLGC regularizer achieved the best results due to two properties. In the first place, it allows the pre-labeled nodes to be altered. This helps to correct errors in the labeling of nodes, improving the classification. In the second place, the algorithm decreases the excessive influence of objects with a high degree in the definition of the information of the nearest classes. This allows that the nodes get a different label from their neighbors.

We compared our best result with the works of Anchiêta and Pardo [3] and Souza and Sanches [10], since they also deal with the paraphrase detection task for Portuguese. Furthermore, we also compared our method with another graph-based method [48]. This method is a Graph Convolutional Network, which contains word nodes and document nodes. The number of nodes in the graph is the number of documents plus the number of unique words in a corpus. The edges among the nodes are based on word occurrence in documents and word co-occurrences in the whole corpus. Moreover, the weight of the edge between a document node and a word node is the Term Frequency-Inverse Document Frequency (TF-IDF) and the weight between two word nodes is the Pointwise Mutual Information (PMI) [49] value. Equation (3) summarizes the approach to weight an edge between nodes.

$$W_{i,j}=\left\{\begin{array}{cc}\mathrm{PMI}(i,j)\hfill & i,j\mathrm{are}\mathrm{words},\mathrm{PMI}(i,j)>0\hfill \\ {\mathrm{TF}\text{-}\mathrm{IDF}}_{i,j}\hfill & i\mathrm{is}\mathrm{document},j\mathrm{is}\mathrm{word}\hfill \\ 1\hfill & i=j\hfill \\ 0\hfill & \mathrm{otherwiese}\hfill \end{array}\right.$$

(3)

In

Table 8. Comparison among approaches on the balanced ASSIN corpus.

Method	F-score		Accuracy
	Paraphrase	Non-Paraphrase	-
Souza and Sanches [10]	0.45	0.55	0.50
Anchiêta and Pardo [3]	0.67	0.89	0.83
Yao et al. [48]	0.68	0.88	0.83
Ours	0.74	0.91	0.87

, we present the results of the comparison between the models, and, as we can see, our strategy outperformed the other methods, achieving better results than other models with only 30% of the pre-labeled data. It is important to say that we trained and evaluated these models on the balanced ASSIN corpus.

We further assessed whether the trained models on the balanced ASSIN corpus improve the results of the models when evaluated on the ASSIN corpus without balancing, i.e., we are interested in check if the data-augmentation strategy used to balance the ASSIN corpus contributes to improve the results when tested on the original ASSIN corpus. The results of this investigation are shown in

Table 9. Comparison among approaches on the ASSIN corpus.

Method	Data-Augmentation	F-Score
		Paraphrase	Non-Paraphrase
Souza and Sanches [10]	✗	0.22	0.77
	✓	0.28	0.80
Anchiêta and Pardo [3]	✗	0.32	0.88
	✓	0.35	0.92
Yao et al. [48]	✗	0.00	0.89
	✓	0.02	0.93
Ours	✗	0.00	0.90
	✓	0.07	0.98

.

One can see that all the models improved their results when trained on the ASSIN corpus with data augmentation, showing that the back-translation strategy for the paraphrase detection task is feasible to produce better models. For this experiment, the method of Anchiêta and Pardo [3] reached the best results, on average. Also, the graph-based methods performed poorly, having difficulty to correctly predict a label with very few instances. ASSIN corpus has 239 instances as paraphrases and 3761 instances as non-paraphrases in the test set, as depicted in Table 4. To alleviate the difficulty of graph-based models to predict a label with very few instances, one may use boosting strategies, as RUSboost [50]. To investigate other approaches to tackle this subtlety remains for future work.

The balanced corpus and our graph-based model are available at (https://github.com/RogerFig/graph-paraphrase).

6. Final Remarks

In this paper, we presented a graph-based approach to model the paraphrase detection task. We defined a heterogeneous network with two semantic node types; sentence pairs and tokens. We created undirected and unweighted links among the token node and sentence pair node types. More than the method, we detailed a data-augmentation strategy, using back-translation to balance the dataset. Our approach outperformed the best results reported for paraphrase detection in Portuguese on the balanced ASSIN corpus both on accuracy and F-score measures.

The defined network is flexible and may be adapted to include other node types, as the embeddings of the sentences or tokens, for example. In addition, because of this flexibility, other network topologies may be explored, as creating weighted links between sentence pair nodes. Furthermore, heterogeneous networks may be applied to a broad range of other NLP tasks, as summarization, dependency parsing, sentiment classification, automatic essay scoring, and others. Another interesting future work is to verify which paraphrase types the systems detect. For that, one could follow the work of Kovatchev et al. [51] to annotate the paraphrase types that occur in the corpus. With this additional annotation layer, it may be possible to perform a qualitative evaluation, helping to explain which paraphrase types the models identify.