Mask Transformer: Unpaired Text Style Transfer Based on Masked Language

Abstract

Currently, most text style transfer methods encode the text into a style-independent latent representation and decode it into new sentences with the target style. Due to the limitation of the latent representation, previous works can hardly get satisfactory target style sentence especially in terms of semantic remaining of the original sentence. We propose a “Mask and Generation” structure, which can obtain an explicit representation of the content of original sentence and generate the target sentence with a transformer. This explicit representation is a masked text that masks the words with the strong style attribute in the sentence. Therefore, it can preserve most of the semantic meaning of the original sentence. In addition, as it is the input of the generator, it also simplified this process compared to the current work who generate the target sentence from scratch. As the explicit representation is readable and the model has better interpretability, we can clearly know which words changed and why the words changed. We evaluate our model on two review datasets with quantitative, qualitative, and human evaluations. The experimental results show that our model generally outperform other methods in terms of transfer accuracy and content preservation.

1. Introduction

The text style transfer task aims to change the stylistic attributes of sentences (e.g., emotions), while retaining the style-independent content of the context as much as possible. This method can be widely used to transfer review sentiment, rewrite news, change dialogue emotion, and so on. For example, a positive comment “The restaurant’s dishes are very delicious, I highly recommend it!” can be transformed into a negative comment “The restaurant’s dishes are a bit disappointing, and I will never come again!” Currently, there is no specific and common definition of text style, so we usually set the definition depending on the task. In addition, due to the high construction costs of parallel corpus, the current research is mainly conducted using nonparallel corpora [1].

Currently, a deep neural network model based on the seq2seq framework is the main text style transfer method. The first class of methods focuses on disentangling the content and style in the latent space and encodes sentences into latent representations in the semantic space and style space, respectively. After disentangling the content, the style-independent representation will be decoded into a sentence with the target style via a generative model [2,3]. Another class of methods attempts to learn the mixed content and style distribution in the latent space and directly map it in the latent space to complete the style transfer [4,5].

Both classes of methods have certain problems. In the first class of methods, the quality of the separated hidden vector is difficult to evaluate, and the latent representation has difficulties to keep rich semantic information of the original sentence due to its limited capabilities, especially for long text. Regarding the direct transfer method, because of the limitation of the latent space, it also cannot keep the rich semantic information of the original sentence. In addition, because of its poor interpretability, it has difficulties dealing with some adaption problems or incorporating external information. In addition, most previous work used recurrent neural networks (RNNs) as encoders and decoders [6], which are limited by their weak abilities to capture long-term dependencies. In summary, previous works can hardly get satisfactory target style sentence especially in terms of semantic remaining of the original sentence.

In this paper, we propose a “Mask and Generation” structure, which can get an explicit representation of the content of original sentence and generate the target sentence with a transformer. The overview of our approach is shown in Figure 1. Our method consists of two parts. In the mask part, we locate the words with higher style attributes in the sentence and replace them with the mask symbols. We follow [7] to train a self-attention [8] style classifier [9] in which the learned attention weights can be used to analyze the style attribute of the words in the sentences. The larger the weight is, the stronger the style attribute. Using this attention mechanism and the style dictionary method, we can find the words with strong style attributes in the sentence. Then, we mask those words to turn the sentence into a neutral one. In the generation part, we take the sentence with the mask symbols as input and use the powerful self-attention mechanism of transformer to generate new sentence according to the target style. Different from [10], who filled [11] the mask position in the sentence, we generate a new sentence with the target style by training a transformer-based [12] generation model for the input masked sentence and the specified style. This approach finds an explicit neutral representation of the sentence via the mask, and then, it flexibly generates the sentence with the target style through the transformer. This not only ensures that the model can capture rich semantic information for the conversion but also maintains high style transfer accuracy.

Our contributions are summarized as follows:

• We propose a “Mask and Generation” structure, which can get an explicit representation of the content of original sentence and generate the target sentence with a transformer. This explicit representation is a masked text that masks the words with the strong style attribute in the sentence. Therefore, it can preserve most of the semantic meaning of the original sentence and also simplify the generator process.
• As the explicit representation is readable, the model has better interpretability, we can clearly know which words changed and why the words changed.
• We use a self-attention mechanism and the style dictionary method to analyze the contributions of the style attributes for each component in the sentence.
• We generate the target sentence by training a transformer-based generation model for the input masked sentence and the specified style.

The experimental results show that our method generally outperforms most models, especially in style accuracy and content preservation.

2. Related Work

In the early stages of the research, text style transfer methods tended to find the style and content representations of sentences. Reference [13] propose a cross-aligned autoencoder with adversarial training [14] to learn a shared latent content distribution and a separated latent style distribution. Reference [15] focused on disentangling style and content representations. They designed a multitask and adversarial loss for a variational autoencoder to ensure the separation. In addition, there are some methods that implicitly find the neutral representation of a sentence. Reference [16] believe that text style transfer can be accomplished using a combination of delete, modify, and generate operations. This is done by deleting words with strong style attributes to find the style-independent part of the sentence and generating new sentences using modifying and generating operations. Reference [10] designed a two-step “Mask and Infill” approach by masking sentimental tokens and predicting words according to the target sentiment.

Although separating the style and content in a sentence will help us in the text style transfer task, it makes it hard to guarantee the quality of the separated style distribution and content distribution. In addition, the fixed-size latent representation limits the generation ability of the model, and it inevitably loses some content information from the original sentence. Therefore, some researchers have proposed some methods that do not need to separate the style and content. Reference [7] was inspired by the cycle style transfer method [17] in Computer Vision. They proposed a cycle reinforcement learning method for nonparallel sentiment transfer tasks. Reference [18] utilized the powerful attention mechanism in the transformer to implement the direct transfer of text style. Reference [19] propose a Context-Aware Style Transfer (CAST) model, which uses two separate encoders for each input sentence and its surrounding context. Reference [20] define a generative probabilistic model that treats a non-parallel corpus in two domains as a partially observed parallel corpus.

The direct transfer method also loses the semantic meaning of the original sentence for the limitation of the latent space. In addition, because of its poor interpretability, it has difficulties dealing with some adaption problems or incorporating external information. Besides, most previous work used recurrent neural networks (RNNs) as encoders and decoders [6], which are limited by their weak abilities to capture long-term dependencies. In summary, previous works can hardly get satisfactory target style sentence especially in terms of semantic remaining of the original sentence.

Based on previous work, our method explicitly obtains a sentence’s neutral representation through masking the strong style attribute words in the sentence. This method greatly enhances the content remaining of the original sentence as well as the interpretability of style transfer. Different from the previous work that directly predicts the masked words, we use the powerful attention mechanism in the transformer to regenerate the neutral masked sentences according to the target style.

3. Approach

In this section, we will introduce our text style transfer method in detail. Section 3.1 is the basic definition of the problem, Section 3.2 is an overview of our model, and Section 3.3, Section 3.4 and Section 3.5 will show the details of the modules one by one.

3.1. Problem Formalization

In this paper, we define the text style transfer problem as follows. Consider a collection of datasets ${\left\{D_i\right\}}_{i=1}^K$, and each dataset $D_i$ consists of many natural language sentences. For all sentences in a single dataset $D_i$, they have some common specific characteristics (for example, they are all positive reviews of a specific product), and we call such shared characteristics the styles of these sentences. In other words, the style is defined by the distribution of the dataset. Suppose we have K different datasets $D_i$. Then, we can define K different styles, and each style is represented by the symbol $s^{\left(i\right)}$. The goal of text style transfer is the following: given a sentence x of any style and the target style $\widehat{s}\in {\left\{s^i\right\}}_{i=1}^K$, rewrite $x$ as a new sentence $\widehat{x}$ with style $\widehat{s}$ and retain the content as much as possible.

3.2. Model Overview

To solve the style transfer problem defined above, our goal is to learn a model with $\left(\widehat{s},x\right)$ as the input, where $x$ is the sentence with the original style attribute, $\widehat{s}$ is the target style, and the output of the model is sentence $\widehat{x}$ with style $\widehat{s}$.

Our method consists of three parts: a Masker module, a Style Generator module, and a discriminator module, as shown in Figure 2. In Section 3.3, the Masker module combines the advantages of the two methods by using the self-attention classification model and the auxiliary style dictionary to find the words with strong style attributes in the sentence. After that, it will perform the mask operation to obtain the masked sentence. In Section 3.4, we take the masked sentence and the specified target style as the input, and the Style Generator will regenerate a sentence with the target style. Unlike the method of filling the mask position, our method will regenerate a complete sentence based on the masked sentence, so that the generated sentence is more flexible in terms of structure and semantics. In addition, a major problem in text style transfer is that there are not enough parallel corpora. Therefore, we cannot directly train our style conversion model in a supervised manner. In Section 3.5, we will introduce a discriminator-based approach [21,22] to conducting supervised training using nonparallel corpora. Finally, we will combine these three parts to train our style transfer model through the learning algorithm in Section 3.6.

3.3. Masker

We first introduce the method based on the self-attention classification model and then introduce an assisted style dictionary method. Finally, we propose a fusion method that combines the advantages of both methods.

3.3.1. Self-Attention Classifier-Based Method

For a sentence, each word in the sentence contributes differently to the sentence’s style. If a word or phrase contributes more to the style of a sentence, it means that this component has a higher style attribute. In other words, if we can find the components with large style contributions in each sentence and mask them, we can obtain masked sentences that tend to be neutral. This is an approach to approximately obtain style-independent representations. For a sentence $x=\langle t_1,t_2,\dots ,t_N\rangle$ with $N$ words, we use bidirectional LSTM (long short-term memory) to encode the sentence and concatenate the forward hidden state and backward hidden state of each word to obtain the final hidden state:

$$H=\left(h_1,h_2,\dots ,h_N\right),$$

(1)

where $N$ is the length of the given sentence. The self-attention mechanism calculates an attention weight vector $a$. In addition, the weighted hidden state vector $c$ is obtained by multiplying the hidden weight vector $H$ by $a$. Finally, we convert $c$ into a probability distribution $y$ through the Softmax layer:

$$a=softmax\left(w\cdot \tanh \left(W{H}^T\right)\right),$$

(2)

$$y=softmax\left(W^{\prime }\cdot c\right),$$

(3)

where $w$, $W$, and $W^{\prime }$ are the network parameters. $W$ maps the hidden layer vector to a high-dimensional space to learn the impact of the input on the label. $w$ is used to map the vector to a scalar, and the attention weight is obtained by Softmaxing the sequence. $W^{\prime }$ maps the output vector after the weighted summation to the category dimension and obtains the probability of each category through Softmax. After sufficient training, the classifier can achieve $97\%$ accuracy. We use $cl{s}_{\rho }$ to represent the attention-based classification model, where $\rho$ represents the model parameters.

Definitely, the attention value calculated during classification can be extracted and used to analyze the contribution of the style attributes for each component of the sentence. Considering that the length of each sentence may be different, using an average attention value or fixed value as a threshold [7] has certain limitations and cannot be adapted to each sentence. We propose a mask method based on the proportion of the sentence length. For example, the words of a sentence are sorted according to their attention values, and then the top $15\%$ of the words will be masked. This strategy can adapt to various sentence lengths.

3.3.2. Style-Dictionary-Based Method

We define this method as follows. For any dataset ${\left\{D_i\right\}}_{i=1}^K$, let $count\left(u,D_i\right)$ denote the number of times the n-gram word $u$ appears in dataset $D_i$. The smoothed frequency ratio represents the significance of $u$ relative to the dataset $D_i$ and is calculated as follows:

$$score\left(u,i\right)=\frac{count\left(u,D_i\right)+\lambda }{({{\displaystyle \sum }}_{i^{*}\in K,i^{*}\ne i}count\left(u,D_{i^{*}}\right))+\lambda },$$

(4)

where $\lambda$ is the smoothing parameter. When $score\left(u,i\right)$ is greater than the threshold ${\gamma }_{score}$, the word $u$ is considered to be a component with a large contribution to the style attribute. Finally, these words form the auxiliary style dictionary $v_i$.

3.3.3. Fusion Method

In this paper, we propose a fusion method that combines the advantages of these two approaches described above. The auxiliary-style-dictionary based method has the advantages of stability and scalability. When the threshold ${\gamma }_{score}$ is larger, the words in style dictionary $v_i$ will be more characteristic. The self-attention classifier-based method has the advantages of flexibility and self-adaptation. It can automatically analyze the style attributes of each component of the sentence and find some components with potential style attributes.

The specific process is as follows. First, calculate the style dictionary $v_i$ through (4). For each sample, mask all the words appearing in the style dictionary $v_i$. If the words masked by the assisted style dictionary method do not reach the mask proportion, the self-attention classifier-based method will remask the sentences by calculating the attention value of each component. For sample $x$ in dataset $D$, we use $\tilde{x}$ to represent the masked sentences—that is, $\tilde{x}=mask\left(x\right)$—and the dataset of masked sentences is denoted as $D_{mask}$.

3.4. Style Generator

In the Style Generator, we chose the standard Transformer model, following the classic encoder-decoder structure. For example, for the input $\tilde{x}=\left(x_1,x_2,\dots ,x_N\right)$, the transformer encoder $Encode{r}_{\theta }\left(\tilde{x}\right)$ maps it to a latent continuous representation vector $c=\left(c_1,c_2,\dots ,c_N\right)$. Then, the transformer decoder $Decode{r}_{\theta }\left(c\right)$ generates the conditional probability of output $y=\left(y_1,y_2,\dots ,y_N\right)$ through an autoregressive calculation as follows:

$$P_{\theta }\left(y|\tilde{x}\right)={{\displaystyle \prod }}_{t=1}^m{P}_{\theta }(y_t|c,y_1,\dots ,y_{t-1}),$$

(5)

For each time $t$, in the decoder, the probability of generating a word is calculated by a Softmax layer:

$$P_{\theta }(y_t|c,y_1,\dots ,y_{t-1})=softmax\left(o_t\right),$$

(6)

where $o_t$ is the logit vector output by the decoder.

To apply the target style control to the generation, we additionally add a mark of the target style before the input, similar to the <cls> mark for BERT. That is, $Encode{r}_{\theta }\left(s,\tilde{x}\right)$. Therefore, the model can calculate the output probability under the conditions of input $\tilde{x}$ and target style s:

$$P_{\theta }\left(y|s,\tilde{x}\right)={{\displaystyle \prod }}_{t=1}^m{P}_{\theta }(y_t|c,y_1,\dots ,y_{t-1}),$$

(7)

We denote the Style Generator model as $f_{\theta }$, where $\theta$ represents the model parameters. Then, the predicted sentence calculated above is denoted by $f_{\theta }\left(s,\tilde{x}\right)$.

3.5. Discriminator

The purpose of introducing the discriminator module is to solve the problem of nonparallel corpus training. When we take samples from the dataset $D$ and obtain $\left(s,\tilde{x}\right)$ through the masker module, but due to the lack of a parallel corpus, we cannot obtain the corresponding reference to sentence $f_{\theta }\left(\widehat{s},\tilde{x}\right)$, while target style $\widehat{s}\ne s$. Therefore, we introduce a discriminator module to learn via supervised training from nonparallel corpora.

For the data $\left(s,\tilde{x}\right)$, we can intuitively restore $\tilde{x}$ to sentence $x$ according to its original style $s$. Furthermore, we use its own supervised training to make the model have a certain style transfer ability. For the target style $\widehat{s}\ne s$, we train a discriminator network to constrain the optimization direction of the generation module in order to better generate target style sentences.

The discriminator network we use includes a Transformer encoder, which is used to distinguish the styles of sentences. The style control information of the discriminator network will be passed to the generation module. Different from the traditional discriminator, in order to better guide the generation module during training, we refer to the discriminator training method of [18] and use two different discriminator structures. We denote the discriminant model as $d_{\phi }$, where $\phi$ is the model parameter.

• Conditional Discriminator

Similar to the discriminator in conditional GANs (Generative Adversarial Networks), the conditional discriminator makes decisions based on the input sentence and style. Specifically, the conditional discriminator $d_{\phi }$ needs to complete a binary classification task, and its inputs are the sentence x and the matching style $s$. The output of the discriminator $d_{\phi }\left(s,x\right)$ determines whether the style of the input sentence $x$ is $s$.

In the discriminator training process, for the style $s$, the positive sample is the real sentence $x$ and the reconstructed sentence $f_{\theta }\left(s,\tilde{x}\right)$, and the negative sample is the transfer sentence $f_{\theta }\left(\widehat{s},\tilde{x}\right),$ while the target style $\widehat{s}\ne s$. In the training process of the Style Generator, the goal of the generator $f_{\theta }$ is to maximize the probability that the discriminator determines that $d_{\phi }\left(\widehat{s},f_{\theta }\left(\widehat{s},\tilde{x}\right)\right)$ is true.

2.

Multi-Class Discriminator

Compared to the former, the multiclass discriminator only uses one sentence as its input, and its goal is to judge the style of the sentence. Unlike traditional discriminators, for K-style tasks, multiclass discriminators need to perform K + 1 classification tasks. The first K categories are K styles, and the last category is the transfer sentence $f_{\theta }\left(\widehat{s},\tilde{x}\right)$ of the target style $\widehat{s}\ne s$. The purpose of this design is to help the generation module learn more accurate knowledge from the discriminator. As the transfer sentence $f_{\theta }\left(\widehat{s},\tilde{x}\right)$ is usually poor at the beginning of training, setting these sentences as another class can make the generator closer to the distribution of real sentences during the iterative training process.

In the discriminator training process, the real sentence $x$ and the reconstructed sentence $f_{\theta }\left(\widehat{s},\tilde{x}\right)$ will be labeled as style s, and the transfer sentence $f_{\theta }\left(\widehat{s},\tilde{x}\right)$ will be labeled as class 0. In the training process of the Style Generator, the goal of $f_{\theta }$ is to maximize the probability that the discriminator determines that $f_{\theta }\left(\widehat{s},\tilde{x}\right)$ has style $\widehat{s}$.

3.6. Training Algorithm

This section will mainly introduce the training algorithm of each module.

• Masker Training Algorithm

The training algorithm of the masker module mainly trains a classification model based on self-attention. Its goal is to determine the style category of each sentence to obtain the attention weight as the subsidiary product. This is the basis for analyzing and masking the sentence. The loss function for the Masker is the cross-entropy loss of the classification problem; that is,

$${ℒ}_{masker}\left(\rho \right)=-{{\displaystyle \sum }}_{\left(s,x_i\right)\in D}^{n}logp(c|x_i),$$

(8)

where dataset $D$ is the original training set.

The learning algorithm of the discriminator mainly trains a classification model based on the Transformer encoder. Its goal is to distinguish between the original sentence $x$, the reconstructed sentence $f_{\theta }\left(s,\tilde{x}\right),$ and the transfer sentence $f_{\theta }\left(\widehat{s},\tilde{x}\right)$. The loss function for the discriminator is the cross-entropy loss of the classification problem.

For the conditional discriminator,

$${ℒ}_{discriminator}\left(\phi \right)=-{{\displaystyle \sum }}_{\left(s,x_i\right)\in D_{dis}}^{n}logp(c|s,x_i),$$

(9)

and for multiclass discriminator,

$${ℒ}_{discriminator}\left(\phi \right)=-{{\displaystyle \sum }}_{\left(s,x_i\right)\in D_{dis}}^{n}logp(c|x_i),$$

(10)

where dataset $D$ consists of $x$, $f_{\theta }\left(s,\tilde{x}\right)$ and $f_{\theta }\left(\widehat{s},\tilde{x}\right)$. The details are given in Algorithm 1.

Algorithm 1: The Training of the Discriminator
	Input: dataset $D_{mask}$, discriminator $d_{\phi }$, style generator $f_{\theta }$
1	for each sentence $\left(s,x,\tilde{x}\right)$ in dataset $D_{mask}$ do
2	Select a style $\widehat{s}$ where $\widehat{s}\ne s$;
3	$y=f_{\theta }\left(s,\tilde{x}\right)$;
4	$\widehat{y}=f_{\theta }\left(\widehat{s},\tilde{x}\right)$;
5	Append $\left(s,x,y,\widehat{s,}\widehat{y}\right)$ to dataset $D_{dis}$
6	end for
7	for each iteration i = 1, 2, …, m do
8	Sample a minibatch of cases from $D_{dis}$;
9	for each case $\left(s,x,y,\widehat{s,}\widehat{y}\right)$ in a batch do
10	if discriminator is multi-class then
11	Set {$x,y$} as class s + 1;
12	Set {$\widehat{y}$} as class 0;
13	Compute ${ℒ}_{discriminator}$ (10);
14	else
15	Set {$\left(s,x\right)$, $\left(s,y\right)$} as True;
16	Set {$\left(\widehat{s},x\right)$, $\left(\widehat{s},\widehat{y}\right)$} as False;
17	Compute $L_{discriminator}$ (9);
18	end if
19	end for
20	Update the model parameters $\phi$;
21	end for

2.

Style Generator Training Algorithm

The training of the style generator is divided into two parts. One part is the case where the target style $\widehat{s}=s$, and the other part is the case where the target style $\widehat{s}\ne s$.

Sentence reconstruction: For the case when the target style $\widehat{s}=s$, we can directly apply a training method that reconstructs the mask sentence $\tilde{x}$ into the original sentence $x$ by using its own supervision information. Specifically, when using the style $s$ and the masked sentence $\tilde{x}$ as input, the model output is as $f_{\theta }\left(s,\tilde{x}\right)$ close as possible to the original sentence $x$. The training goal is to minimize the negative log-likelihood:

$${ℒ}_{reconstruction}\left(\theta \right)=-{{\displaystyle \sum }}_{\left(s,\widetilde{x_i}\right)\in D_{mask}}^{n}logp(y=x|s,{\tilde{x}}_i),$$

(11)

where the dataset $D_{mask}$ is obtained by masking the sentences in the original training set $D$.

Style generation: For the case when the target style $\widehat{s}\ne s$, we introduce a loss function to control the generation of the style, so that the transformed sentences are closer to the distribution of the real sentences and the reconstructed sentences.

Using the conditional discriminator, we can obtain the probability that $d_{\phi }(\widehat{s},f_{\theta }\left(\widehat{s},\tilde{x}\right)$ is true. The goal of the generator is to minimize the negative log-likelihood:

$${ℒ}_{style}\left(\theta \right)=-{{\displaystyle \sum }}_{\left(s,\widetilde{x_i}\right)\in D_{mask}}^{n}logp(c=1|f_{\theta }\left(\widehat{s},\tilde{x}\right),\widehat{s}),$$

(12)

Using the multiclass discriminator, we can calculate the probability that $d_{\phi }\left(f_{\theta }\left(\widehat{s},\tilde{x}\right)\right)$ has the style $\widehat{s}$. The goal of the generator is to minimize the negative log-likelihood of the class probability:

$${ℒ}_{style}\left(\theta \right)=-{{\displaystyle \sum }}_{\left(s,\widetilde{x_i}\right)\in D_{mask}}^{n}logp(c=\widehat{s}|f_{\theta }\left(\widehat{s},\tilde{x}\right)),$$

(13)

Combining the loss functions described above, we can conduct training for the generator. Algorithm 2 shows the details of the training process.

Algorithm 2: The Training of the Style Generator
	Input: dataset $D_{mask}$, discriminator $d_{\phi }$, style generator $f_{\theta }$
1	for each iteration i = 1, 2, …, m do
2	Sample a minibatch of cases from $D_{mask}$;
3	for each case $\left(s,x,\tilde{x}\right)$ in a batch do
4	Select a style $\widehat{s}$ where $s^\ne s$;
5	Reconstruct the sentence $y=f_{\theta }\left(s,\tilde{x}\right)$;
6	Generate a sentence $\widehat{y}=f_{\theta }\left(\widehat{s},\tilde{x}\right)$;
7	Judge $\widehat{y}$ by discriminator $d_{\phi }$;
8	Compute $L_{reconstruction}$ (11);
9	Compute $L_{style}$ (12 or 13);
10	end for
11	Update the model parameters $\theta$;
12	end for

3.

Summarization

By combining the training algorithms of the above modules, we can obtain the whole training process of the model. In the mask part, first, the self-attention model of the Masker is trained. After the model convergence is stable, the Masker masks the sentences in the original dataset $D$ and obtains the dataset $D_{mask}$. In the style generation part, using the alternate training method of the generator and discriminator in GANs [14], we also alternately train the Style Generator and Discriminator. In each iterative round, we first train the Discriminator $n_d$ steps to obtain an optimized discrimination module. Under the updated Discriminator, the Style Generator will be trained $n_g$ steps to optimize the results of the generation. The iterative training continues until the model converges and stabilizes. The specific algorithm is given in Algorithm 3.

Algorithm 3: The Whole Training Algorithm
	Input: dataset $D$, attention-based classifier $cl{s}_{\rho }$
1	for each iteration $i=1,2,\dots ,m$ do
2	Sample a minibatch of sentences from $D$;
3	Compute ${ℒ}_{masker}$ (8) for the batch;
4	Update the model parameters $\rho$;
5	end for
6	Build the style dictionary by (5)
7	for each sentence in dataset $D$ do
8	Mask the sentence using the style dictionary;
9	if (mask words/length of sentence) < masking percentage then
10	Mask again according to the attention weight from clsρ;
11	end if
12	Append the case $\left(s,x,\tilde{x}\right)$ into $D_{mask}$;
13	end for
14	Initialize Style Generator network $\theta$ and Discriminator network $\phi$;
15	for each iteration i = 1, 2, …, m do
16	for $n_d$ steps do
17	Algorithm 2;
18	end for
19	for $n_g$ steps do
20	Algorithm 3;
21	end for
22	end for

The learning algorithm of the discriminator mainly trains a classification model based on the Transformer encoder. Its goal is to distinguish between the original sentence $x$, the reconstructed sentence $f_{\theta }\left(s,\tilde{x}\right)$ and the transfer sentence $f_{\theta }\left(\widehat{s},\tilde{x}\right)$. The loss function for the discriminator is the cross-entropy loss of the classification problem.

In addition, there is a problem in training that needs to be briefly explained. Due to the discrete nature of natural language, when we obtain the transfer sentence and input it into the discrimination module, the gradient calculated by the discrimination module cannot be propagated back to the generation module. To solve this problem, it is common to use the Gumbel–Softmax strategy or the reinforcement learning method to evaluate the gradient from the discriminator. However, both methods have the problem of high variance, which makes it difficult for the model to converge and stabilize. Therefore, we use the way that [18] deal with discrete sample problems. Instead of directly using the generated words as the input, we use the Softmax distribution generated by $f_{\theta }$ as the input. Similarly, for the decoder of the generator, the decoding method is also changed from greedy decoding to continuous decoding. Specifically, at each calculation time, instead of using the word with the highest probability predicted in the previous step, we use the probability distribution as the input. Regarding the input in the form of a probability distribution, the decoder will calculate the weighted average representation of the probability distribution through the embedding matrix.

4. Experiment

4.1. Datasets

We compared our method with several recent approaches on two review datasets: the Yelp Review dataset (Yelp) and the IMDb Movie Review dataset (IMDb).

Table 1. The statistics of the Yelp and IMDb datasets.

Dataset	Attributes	Train	Dev	Test	Avg Len
Yelp	Positive	266k	2000	500	8.9
	Negative	177k	2000	500
IMDb	Positive	179k	2000	1000	18.5
	Negative	188k	2000	1000

shows the statistics of these two datasets.

Yelp Review dataset (Yelp): The Yelp dataset is provided by the Yelp dataset challenge. It consists of restaurant and business reviews with sentiment labels (negative or positive). Following previous work, we used the dataset provided by [16]. In addition, it also provides artificial reference sentences for the test set

IMDb Movie Review dataset (IMDb): The IMDb dataset consists of movie reviews written by online users. To obtain a high-quality dataset, we used the highly polar movie reviews provided by [18]. Using the original dataset, they construct a highly polar sentence-level style transfer dataset via the following steps. (1) A BERT-based classifier is trained and fine-tuned on the original dataset. The accuracy is 95%. (2) The original comment in the dataset is split into several sentences. (3) Each sentence with a confidence threshold lower than 0.9 is filtered by the fine-tuned BERT-based classifier. (4) Sentences containing uncommon words are deleted. Finally, the highly polar dataset contains 366k training, 4k validation, and 2k test sentences.

4.2. Evaluation

An ideal transfer sentence should be prominent, content-complete, and fluent. After referring to the evaluations in previous work, we mainly focus on the following three aspects of sentence generation: (1) the degree of style transfer, (2) content preservation, and (3) fluency. In terms of the current evaluation methods, we include two parts: automatic evaluation and manual evaluation.

4.2.1. Automatic Evaluation

Style transfer: To evaluate the style transfer, we use the accuracy of the generated sentences in the style classification as an automatic evaluation metric. Specifically, we refer to [9] and use fastText to train a style classifier based on the Yelp and IMDb datasets, respectively. For the transfer sentence $f_{\theta }\left(\widehat{s},\tilde{x}\right)$, we use the classifier to evaluate the style accuracy.

Content preservation: To measure the content preservation, we adopt the BLEU (Bilingual Evaluation Understudy) score [23] as the evaluation metric. Specifically, we use the calculation tool provided by NLTK (Natural Language Toolkit) to calculate the BLEU score for the transfer sentence and the original sentence. A high BLEU score indicates that there is a high degree of similarity between the converted sentence and the original sentence on the word-level. This indicates that there is good content reservation. In addition, if the dataset provides artificial reference sentences, we will also calculate the BLEU score of the transfer sentence and the reference sentence. The two BLEU metrics are defined as self-BLEU and ref-BLEU.

Fluency: A common way to evaluate the fluency is calculating the perplexity of the transfer sentence. Specifically, we use KenLM to train a 5-g language model on the Yelp and IMDb datasets, and we use this model to calculate the perplexity of the sentence. The lower the perplexity is, the higher the generation probability and fluency of the sentence.

4.2.2. Manual Evaluation

For the manual evaluation, we chose the scoring method and hired three reviewers to score the output of our model and the best models of [18] (Style Transformer) and [16] (Delete and Retrieve). Scoring is similar to the automatic evaluation. We mainly evaluate three aspects: the style transfer, the content reservation, and the fluency of the output results. Our scoring scale has scores that range from 1 (very poor) to 5 (very good). For each dataset, we will sample 100 sentences for each target style in the evaluation.

4.3. Training Details

Mask part: For the self-attention classification model, the word embedding size is 256, the bidirectional LSTM hidden size is 256, and the number of hidden layers is two. For training, we use the Adam optimizer with a learning rate of 0.001 and train for five epochs. For the assisted style dictionary method, we use the statistical results of 1-g and 2-g, set the smoothing parameter $\lambda$ to 1, and set the threshold ${\gamma }_{score}$ to 0.75.

Generation part: In the generator and the discriminator, we use four layers of transformers, and each layer has a multi-head attention of eight. The word embedding, position encoding, and style embedding are all 256 in size. In the encoder, style embedding will be added to the head of the sentence as a mark, and the mark does not use position encoding information. For the discriminator, similar to the BERT [24], we add the $\langle cls\rangle$ mark to the head of the input sentence, and the output corresponding to this mark will be passed to a Softmax layer to obtain the output of the discriminator. In terms of the training parameters, $n_d$ is set to 10 and $n_g$ is set to five. We use the Adam optimizer with a learning rate of 0.0001.

In the experiment, in order to improve the robustness of the model and converge to a more reasonable result, we use the practice of [18]. When the model calculates the reconstruction loss function (6), we conduct random word dropout on the input, which makes the model more robust. The experiments show that the random word dropout improves the transfer results in some cases.

4.4. Experimental Results

Table 2. The automatic evaluation results of the Yelp and IMDb datasets.

	Yelp				IMDb
	ACC	Self-BLEU	ref-BLEU	PPL	ACC	Self-BLEU	PPL
ControlledGen [21]	88.8	45.7	14.3	219	94.6	62.1	143
DeleteOnly [16]	85.7	28.6	9.7	72	N/A	N/A	N/A
RetrieveOnly [16]	92.6	0.7	0.4	7	N/A	N/A	N/A
TemplateBased [16]	84.3	44.1	13.7	117	N/A	N/A	N/A
DeleteAndRetrieve [16]	87.7	29.1	10.4	60	58.8	55.4	57
CycleRL [7]	88.0	7.2	2.8	107	97.8	4.9	177
StyleTransformer [18]	89.5	54.5	18.7	76	82.3	67.5	108
Conditional MaskTransformer	91.8	54.6	19.3	81	95.7	63.9	92
Multi-class MaskTransformer	88.3	55.4	20.1	75	91.0	66.2	86

ACC and PPL denote accuracy and perplexity, respectively.

shows the automatic evaluation results of the model on the two datasets. In the Yelp dataset, although RetrieveOnly achieved the highest accuracy rate of 92.6%, and the perplexity is the lowest of 7. While in terms of semantic retention, the self-BLEU of RetrieveOnly was only 0.7, and the ref-BLEU was only 0.4. Almost no information about the original sentence was retained. This is because RetrieveOnly uses a retrieve method to find the most suitable template in the target style dataset, and directly use the template as the generated sentence. Therefore, the style conversion degree of this method is very high and the fluency of the template sentence almost as natural language. However, since the content of the generated sentence is completely changed compared to the original sentence, this method has no practical value. In terms of IMDB, the CycleRL method suffers from the same problem. Although it achieves an accuracy of 97.8% in style conversion, it only has a self-BLEU of 4.9 and a perplexity of 177 in terms of semantic retention and language fluency. We can see that our model has achieved the best results in style accuracy and content reservation for both datasets, and it also has better perplexity.

For the manual evaluation, we select two representative models: Delete And Retrieve [16] and the Style Transformer [18]. For our method, we chose the model based on the multiclass discriminator. We randomly sample 100 sentences for each style in the dataset, and the transfer sentences were scored by three reviewers after being scrambled. The results are shown in

Table 3. The manual evaluation results of the Yelp and IMDb datasets.

	Yelp			IMDb
	Style	Content	Fluency	Style	Content	Fluency
DeleteAndRetrieve	3.67	2.61	2.84	3.75	2.33	2.26
StyleTransformer	3.41	3.34	3.64	3.36	3.58	3.52
Ours (conditional)	4.12	4.08	3.67	3.98	3.87	3.31

. It can be seen that our model performs significantly better than other methods in style transfer accuracy as well as content remaining and fluency of the target sentence. To better understand the characteristics of our model, we sampled some sentences in the Yelp dataset, as shown in

Table 4. Case studies from the Yelp dataset.

Positive to Negative
Input	the salads were fresh and crispy.
Masked	the salads were [mask] and [mask].
ST	the salads were dry and crispy.
Ours	the salads were terrible and not worthy.
Reference	the salads were old and wilted.
Input	reasonable price, bottom line guaranteed.
Masked	[mask] price, bottom line guaranteed.
ST	reasonable price, bottom line fees.
Ours	overpriced and no bottom line fees.
Reference	the price isn’t reasonable, bottom line isn’t guaranteed.
Negative to Positive
Input	i can’t believe how inconsiderate this pharmacy is.
Masked	i ca [mask] believe how [mask] this pharmacy is.
ST	i can always believe how compassionate this pharmacy is.
Ours	i can always believe how impressed this pharmacy is.
Reference	this pharmacy is really considerate.
Input	the burgers were over cooked to the point the meat was crunchy.
Masked	the burgers were [mask] cooked to the point the meat was [mask].
ST	the burgers are while cooked to the point the meat loved crunchy!
Ours	the burgers were cooked to the point and the meat was delicious.
Reference	the burgers were cooked perfectly and the meat was juicy.

.

5. Conclusions

In this paper, we focus on solving the style transfer problem of nonparallel corpora and propose a style generation method based on a “Mask and Generation” structure, which can be trained using nonparallel corpora and has good interpretability. The experimental results on two review datasets show that our method outperforms previous approaches in terms of style conversion and content reservation. The masked sentences can keep the semantic meaning of the original sentence and also help us to understand the transfer process. In the future, we want to further explore the introduction of prior knowledge in the masker module and improve the model for more fine-grained tasks (multiple emotions).