Generating Scenery Images with Larger Variety According to User Descriptions

Abstract

In this paper, a framework based on generative adversarial networks is proposed to perform nature-scenery generation according to descriptions from the users. The desired place, time and seasons of the generated scenes can be specified with the help of text-to-image generation techniques. The framework improves and modifies the architecture of a generative adversarial network with attention models by adding the imagination models. The proposed attentional and imaginative generative network uses the hidden layer information to initialize the memory cell of the recurrent neural network to produce the desired photos. A data set containing different categories of scenery images is established to train the proposed system. The experiments validate that the proposed method is able to increase the quality and diversity of the generated images compared to the existing method. A possible application of road image generation for data augmentation is also demonstrated in the experimental results.

1. Introduction

Generative adversarial networks (GAN) [1] have become an active research area since they were proposed. The principle of a generative adversarial networks is as follows. The generator generates fake data that serves as negative training samples for the discriminator. The discriminator learns to distinguish the real data from the fake data generated by the generator. Then, the discriminator penalizes the generator for producing implausible results. The generator learns to mimic the real data and generate better results hoping to fool the discriminator. Inspired by game theory, the generator and discriminator complete with each other to achieve the Nash equilibrium in the training process. As both the generator and the discriminator are well-trained, the GAN can produce plausible results. It usually requires a lot of training data and computation time to train a steady GAN model. Due to the weakness of the original GAN, various improved GANs models in terms of architecture optimization and objective function optimization have been proposed [2]. Radford et al. [3] proposed a Deep Convolutional Generative Adversarial Networks (DCGAN) which replaces the fully connected layer with the deconvolution layer in the generator to achieve better performance in image generation tasks. Mirza and Osindero [4] proposed Conditional Generative Adversarial Networks (CGANs), which introduce the conditional variable in both the generator and discriminator so that conditions such as labels, text or other data can be added to control to the model. Chen et al. [5] proposed another CGAN called InfoGAN by introducing mutual information and making the generation process more controllable and more interpretable. Based on CGAN, Odena et al. [6] proposed an Auxiliary Classifier GAN (ACGAN). The condition variable is not added in the discriminator, and another classifier is used to indicate the probability over the class labels. The loss function is also modified to increase the probability of correct prediction for the discriminator.

In the training process of GAN, mode collapse has always been a problem. Sometimes it is feasible to fine tune the training parameters for better model learning. However, tuning parameters might not always work well. Therefore, Arjovsky et al. [7] used Wasserstein distance to improve the stability of learning and mode collapse. They showed that the Earth-Mover distance could produce better gradient behaviors when performing distribution learning compared with other distance metrics. Metz et al. [8] proposed an unrolled GAN that uses a gradient-based loss function to enhance the generator and applies the Jensen–Shannon (JS) divergence to minimize the loss function of the generator. Nguyen et al. [9] proposed a dual discriminator architecture. One of the discriminators rewards high scores for samples from true data distribution and the other discriminator rewards data from the generator, and thus produces data to fool both two discriminators.

The research area of GAN has many applications including art, design, data augmentation, and other applications [10,11,12,13,14]. For example, the research in [5] generates hand-written numbers. The DRAW system [15] constructs license plates. The work in [16] generates anime characters. The system in [3] demonstrates generating bedroom interiors using deep convolutional generative adversarial networks. PixelCNN [17] generates various different animals and human faces. Ledig et al. [18] proposed a super-resolution GAN to take a low-resolution image as input and generate a high-resolution image. Isola et al. [19] proposed an image-to-image translation framework called pix2pix based on CGAN to convert an image content from one domain to another. Iizuka et al. [20] proposed to perform image inpainting using two discriminators to monitor local features and global features for both local and global consistency. The development of the GAN techniques also leads to promising results in some novel applications such as 3D object generation, medicine, and autonomous driving [21]. Yu et al. [22] proposed a 3D object inpainting network that can process point cloud data directly without human labeling. Shin et al. [23] proposed a GAN-based method to generate synthetic, abnormal, MRI images with brain tumors. The work in [24] designed a systematic framework to automatically generate test cases to imitate real-world driving scenes.

With the progress of computer vision and natural language analysis, generating images according to natural language descriptions has become a popular research topic in recent years [25,26,27,28,29,30,31,32]. The work in [26] was one of the first works that attempted to generate images according to text descriptions. It translates single-sentence human-written descriptions directly into image pixels. Images of bird and flower images are synthesized based on human-written texts using a character-level text encoder and class-conditional GAN. It is the first end-to-end differentiable architecture from the character level to pixel level, which introduces a manifold interpolation regularizer for the GAN generator that significantly improves the quality of generated samples. In [27], a semantic compositional network is proposed for image captioning. The network uses a Long Short-Term Memory (LSTM) network and extends each weight matrix of the LSTM to an ensemble of tag-dependent weight matrices. In earlier research, the network architecture did not include training word embeddings. To address the research topic of the relationship between natural language and images, [28] proposed a network to extract visual descriptions that can represent images. To generate high-resolution photo-realistic images from text descriptions, it is not feasible to simply add more upsampling layers in the network. Therefore, StackGAN [29] used two stages of generative networks to first generate a low-resolution image conditioned on the text description, and then refine the results to produce a high-resolution image. Although some good results have been achieved, these studies are only conditioned on global sentence encoding when generating images. Without important fine-grained information at the word level, this degrades the quality of generated images when generating complex scenes. AttnGAN [30] improved the work in [29] by adding embeddings of both word-level and sentence-level descriptions and built an attention model to compute a region-context vector for each word. The work in [32] combined the method of reinforcement learning in the framework to give higher rewards to novel generated images with unseen features and to give lower rewards to repetitive and similar images.

This work focuses on outdoor scenery image generation. To give the generated images more variety, the proposed system improves the architecture of AttnGAN [30] and adds an imagination mechanism in the framework to increase the diversity of the generated scenes. In this way, the variety of the generated images can be effectively increased. The rest of the paper is organized as follows. Section 2 elaborates the details of the proposed framework. Section 3 shows and discusses the experimental results. Finally, conclusions are made in Section 4.

2. Attentional and Imaginative Generative Networks for Scenery Image Generation

The proposed system framework is illustrated in Figure 1. The framework improves and modifies the architecture of AttnGAN [30] by adding the imagination models in addition to the original attention models. Therefore, the core is named as the attentional and imaginative generative networks. The architecture of the original AttnGAN is displayed in Figure A1 in Appendix A for comparison. We can observe that the main difference between the two architectures is the existence of the imagination model, $S_i^{imag}$. The details of the attentional and imaginative generative networks in this work are described below. There are m generators ($G_0, G_1,\dots ,G_{m-1}$) in the generative model. Each of the m generators has its corresponding stage ($S_0,S_1,\dots ,S_{m-1}$) and outputs a hidden layer ($h_0,h_1,\dots ,h_{m-1}$) to generate images of lower to higher resolutions (${\widehat{x}}_0,{\widehat{x}}_1,\dots {\widehat{x}}_{m-1}$) where $\widehat{x_i}=G_i\left(h_i\right)$.

In Figure 1, the text encoder is a bidirectional LSTM [33], which extracts the global sentence vectors $\overline{e}$ from the input text descriptions. $F_{CA}$ is the conditioning augmentation [29] that converts the sentence vector $\overline{e}$ to the conditioning vector c. The vector z is a noise vector sampled from a normal distribution. $S_i$ is the upsampling network at each stage. All stages include attention models $S_i^{attn}$ and imagination models $S_i^{imag}$, except for $S_0$. Note that $F_{CA}, S_i^{imag}, S_i^{attn},S_i$, and $G_i$ are all modeled as neural networks. The relationship of hidden layers $h_i$ and the upsampling networks $S_i$ can be expressed as Equations (1) and (2).

$$h_0=S_0\left(z;F_{CA}\left(\overline{e}\right)\right)$$

(1)

$$h_i=S_i\left(h_{i-1};S_i^{attn}\left(e,h_{i-1}\right)\right)\mathrm{for}\mathrm{i}=1,2,\dots m-1$$

(2)

where $e$ is the matrix of word vectors.

Attention models $S_i^{attn}$ take the word features $e$ and the image features $h_{i-1}$ from the previous hidden layer as the inputs. A new perceptron layer is added to convert the word features $e$ into the common semantic space of the image features, denoted by $e_i^{~}$. Afterwards, a word-context vector $C_j$ is computed for the jth subregion of the image according to its hidden features $h$. Each column of $h$ is a feature vector of the jth subregion. The computation of the word–context vector $C_j$ is expressed in Equations (3) and (4). In Equation (4), the softmax function converts $s_{j,i}^{~}$ into ${\beta }_{j,i}$, which is the weight the attention model learns from the ith word in the process of generating the jth subregion of the generated scene.

$$s_{j,i}^{~}=h_j^T{e}_i^{~}$$

(3)

$$C_j={\displaystyle \sum }_{i=0}^{T-1}{\beta }_{j,i}{e}_i^{~},where{\beta }_{j,i}=\frac{exp\left(s_{j,i}^{~}\right)}{{{\displaystyle \sum }}_{k=0}^{T-1}exp\left(s_{j,k}^{~}\right)}$$

(4)

Imagination models $S_i^{imag}$ are responsible for increasing the variety of the generated images. To elaborate the imagination models, the mechanism of LSTM cells is investigated first. The core concept of an LSTM cell is to add a cell state to enhance the decision in each step. It uses different gates to control the network. The first one is the forget gate, shown in Figure 2a. The inputs of the forget gate are the current input $x_t$ and the previous output $h_{t-1}$. This gate uses a sigmoid function, denoted by $\mathsf{\sigma }$, to control if it should forget or remember the information from the previous cell. The second gate is the input gate as shown in Figure 2b. The input gate determines how much new information should be updated to the new cell. The sigmoid function highlighted in Figure 2b determines the update value $i_t$. Additionally, the tanh function creates the new candidate value ${\widehat{C}}_t$, which is to be added to the state value. When updating the old cell state $C_{t-1}$ into the new cell state $C_t$, the old state is multiplied by the output of the forget gate $f_t$, and then added by $i_t\times {\widehat{C}}_t$, as illustrated in Figure 2c. The output $h_t$ is based on the cell state but being filtered. The old cell state $C_{t-1}$ goes through a tanh function and is multiplied by the output of a sigmoid function. To summarize the mechanism, the forget gate decides what should be remembered. The input gate controls what information is relevant and should be added to the state in the current step. Additionally, the output gate decides what the next hidden state should be.

The imagination model utilizes the LSTM cell in the text encoder. In the upsampling processes of $S_i$, it utilizes the hidden features $h_i$ to reinitialize the cell state in the text encoder. The main difference between the proposed framework and the original framework in [30] is that the imagination model extracts the features again from the text descriptions in every upsampling process. Such design enriches the text features given in the corpus description.

The loss function for the attentional and imaginative generative networks is defined in Equation (5).

$$ℒ={ℒ}_G+{\lambda }_1{ℒ}_C+{\lambda }_2{ℒ}_{DAMSM}$$

(5)

The first term ${ℒ}_G$ in Equation (5) is the summation of the loss at each stage i. We can express it as

$${ℒ}_G={{\displaystyle \sum }}_{i=0}^m{ℒ}_{G_i}$$

(6)

At the ith stage, the adversarial loss for the generator $G_i$ is defined in Equation (7).

$${ℒ}_{G_i}=-\frac{1}{2}{\mathbb{E}}_{{\widehat{x}}_i~p_{G_i}}\left[log\left(D_i\left({\widehat{x}}_i\right)\right)\right]-\frac{1}{2}{\mathbb{E}}_{{\widehat{x}}_i~p_{G_i}}\left[log\left(D_i\left({\widehat{x}}_i,\overline{e}\right)\right)\right]$$

(7)

The first part of Equation (7) is the unconditional loss, which discriminates if the image is real or fake. Additionally, the second part is the conditional loss, which determines whether the image and the text description match or not. In Equation (7), $D_i$ denotes the corresponding discriminator for $G_i$. For each discriminator $D_i$, the cross-entropy loss defined in Equation (8) is minimized to classify the input as real or fake.

$$\begin{gathered} {ℒ}_{D_i}=-\frac{1}{2}{\mathbb{E}}_{x_i~p_{dat{a}_i}}\left[log\left(D_i\left(x_i\right)\right)\right]-\frac{1}{2}{\mathbb{E}}_{{\widehat{x}}_i~p_{dat{a}_i}}\left[log\left(1-D_i\left({\widehat{x}}_i\right)\right)\right] \\ +-\frac{1}{2}{\mathbb{E}}_{x_i~p_{dat{a}_i}}\left[log\left(D_i\left(x_i,\overline{e}\right)\right)\right]-\frac{1}{2}{\mathbb{E}}_{{\widehat{x}}_i~p_{dat{a}_i}}\left[log\left(1-D_i\left({\widehat{x}}_i,\overline{e}\right)\right)\right] \end{gathered}$$

(8)

Here, $x_i$ is from the true image distribution $p_{dat{a}_i}$ at the ith scale, and ${\hat{x}}_i$ is from the model distribution $p_{G_i}$ at the ith scale.

The second term ${ℒ}_C$ in Equation (5) is designed to learn the relationship between the hidden state and image text in the imaginative model $S_i^{imag}$. The relationship $r_h$ between each cell memory state $c_h$ and word vector $e_j$ is defined in the following equation.

$$r_h=c_h\times e_j{}^T$$

(9)

In this way, the relationship between the cell state initialized by the imaginative model $S_i^{imag}$ and the jth word in the word vector can be learned. Then, the cosine similarity of $r_h$ and the sentence vector $s_h$ is calculated using Equation (10).

$$R\left(r_h,s_h\right)=r_h^T{s}_h/\left((\Vert r_h\Vert \Vert s_h\Vert \right)$$

(10)

For each minibatch, we calculate the matching score of the hidden state H and description D.

$$MS\left(H, D\right)=\log {\left({\displaystyle \sum }_{h=0}^{N_h}exp\left({\gamma }_{\mathrm{ms}}R\left(r_h,s_h\right)\right)\right)}^{\frac{1}{{\gamma }_{\mathrm{ms}}}}$$

(11)

In (11), the term $N_h$ is the number of cell memory states, and ${\gamma }_{\mathrm{ms}}$ is a parameter that determines the importance of the pair with the largest similarity $R\left(r_h,s_h\right)$ when calculating the matching score.

Suppose there are $N_{\mathrm{s}}$ samples in a minibatch. We calculate the matching score of the hidden state $H_k$ and text description $D_k$ of each sample k. Then, the probability of description $D_k$ being matched with $H_k$ is defined as Equation (12), where ${\gamma }_{\mathrm{p}}$ is an empirically determined parameter. Finally, the loss term ${ℒ}_c$ for the imaginative model is defined in Equation (13).

$$P\left(H_k|D_k\right)=\frac{exp\left({\gamma }_{\mathrm{p}}R\left(H_k,D_k\right)\right)}{{{\displaystyle \sum }}_{k=1}^{N_{\mathrm{s}}}exp\left({\gamma }_{\mathrm{p}}R\left(H_k,D_k\right)\right)}$$

(12)

$${ℒ}_c=-{\displaystyle \sum }_{i=1}^{N_{\mathrm{s}}}logP\left(H_i|D_i\right)$$

(13)

The third term ${ℒ}_{DAMSM}$ of the loss function in Equation (5) is a word-level image-to-text matching loss. The details of this loss can be found in [30]. Finally, ${\lambda }_1$ and ${\lambda }_2$ in Equation (5) are parameters to balance the three terms in the final loss function.

3. Experimental Results

The experimental results are presented in this section. To train the proposed framework, the training dataset is collected from Open Images Dataset V4 [34] and Places [35]. Open Images Dataset V4 contains 19,995 machine-labeled classes. Each label is provided with a confidence score. We use these labels to select the desired training images for the proposed system. Additionally, images with desired labels are selected from the Places dataset automatically. We select 26 different training descriptions for the Open Images Dataset V4, and 37 different training texts for the Places dataset in the experiments. We refer to the collected dataset set as the Scene dataset. In the Scene dataset, there are total 63 different descriptions. Each description contains 685 images. The total image count is 43,155. To test the proposed system, 1200 images are generated for each training description. Additionally, additional testing descriptions are established by combining different place descriptions randomly with some season or time descriptions in the training texts. There are a total of 76 additional testing description combinations. We also generate 1200 images for each additional testing description. Consequently, 166,800 images are generated to evaluate the image generation system in total. The experiment environment and settings are listed in

Table 1. Experiment Environment and Settings.

Hardware
CPU	i7-8700k
RAM	64GB
GPU	nvidia 1080ti
Environment
OS	Ubuntu 16.04.5 LTS
Docker	18.06.1-ce
Python	python3.5.2
IDE	Vscode1.34.0
Tools
pytorch	1.0.1post2
easydict	1.9
python-dateutil	2.7.2
pandas	0.22.0
torchfile	0.1.0
nltk	3.4
scikit-image	0.14.2
piexif	1.1.2

.

Two metrics, Inception Score (IS) [36] and Fréchet Inception Distance (FID) [37], are used to evaluate the generated images in the experiments. The Inception Score uses an Inception v3 network pretrained on ImageNet. It calculates the statistics of the outputs of the Inception v3 network when applied to generated images. The definition of IS is expressed in Equation (14), where $x~p_g$ denotes that $x$ is an image sampled from the trained generative model.

$$IS\left(G\right)=exp\left({\mathbb{E}}_{x~p_g}{D}_{KL}\left(p\left(y|\mathrm{x}\right)\Vert p\left(y\right)\right)\right)$$

(14)

$$D_{KL}\left(P\Vert Q\right)={\displaystyle \sum }_{i}P\left(i\right)log\frac{P\left(i\right)}{Q\left(i\right)}$$

(15)

$D_{KL}\left(P\Vert Q\right)$ is the KL divergence between the distributions p and q. In Equation (14), $p\left(y|\mathrm{x}\right)$ is the conditional class distribution, and $p\left(y\right)$ is the marginal class distribution. Higher inception scores indicate that the generated images have larger variety, and each image can be distinctly classified as some category. Therefore, the higher the inception score, the better the generative model is.

As indicated in [36], there are some drawbacks to use inception score alone. Fréchet Inception Distance is proposed to remedy the issues of IS. To calculate FID, the inception network is used to extract features from an intermediate layer. The definition of FID between real data ${\mathbb{P}}_r$ and generated data ${\mathbb{P}}_g$ is expressed in Equation (16).

$$\mathrm{FID}\left({\mathbb{P}}_r,{\mathbb{P}}_g\right)={\Vert {\mu }_r-{\mu }_g\Vert }^2+Tr\left(C_r+C_g-2{\left(C_r{C}_g\right)}^{\frac{1}{2}}\right)$$

(16)

In Equation (16), ${\mu }_r$ and ${\mu }_g$ denote the mean of the real data and generated data, respectively. Additionally, $C_r$ and $C_g$ denote the covariance matrices of the real data and generated data, respectively. Finally, Tr sums up all the diagonal elements. Lower FID values indicate better quality and diversity for generated images. FID is more robust to noise than IS, and it is also a better measurement for image diversity.

Figure 3 shows the inception scores of the proposed method and AttnGAN [30] using a different number of training epochs. The number of epochs ranges from 10 to 65 in the training process. Figure 4 compares the FID values of the proposed method and AttnGAN using a different number of training epochs. We can observe that IS and FID do not improve much after number of epochs is larger than 50. Therefore, we select 50 as the number of epochs in the rest of the experiments to save training time. From Figure 3 and Figure 4, we can see that the proposed method can achieve lower FID while maintaining similar or slightly higher IS compared to AttnGAN. Figure 5a displays selected scenery images generated using training descriptions. Images in Figure 5a are generated using the description “forest in the morning”. Figure 5b–d display selected images generated using testing description combinations. Images in Figure 5b are generated using the description “hayfield in autumn morning.” Images in Figure 5c are generated using the description “beach in winter morning,” and those in Figure 5d are generated using the description “sand in autumn.” We can observe that, although images generated under the training descriptions look more realistic, images generated under testing description combinations still exhibit decent quality. This indicates that the model has learned from the word vectors to generate season, time and place combinations that it has never seen in the training images. The limitation of this work is similar to existing GAN-based methods. The model can generate good results if the input is mapped into the learned subspace. However, data that is unseen would be mapped out of the subspace. For example, since a baseball field is not in the training dataset, text descriptions containing baseball fields could not be generated properly.

Figure 6 compares some images generated by the proposed method and AttnGAN. We use the same description to generate 12 images at a time using both methods to observe the differences. Figure 6a,b shows images generated under the description “canal at night.” Figure 6c,d displays images generated under the description “canal in autumn.” Images in Figure 6a,c are generated by AttnGAN. Additionally, images in Figure 6b,d are generated by the proposed method. We can see that similar images appear repeatedly in Figure 6a,c, and the images in Figure 6b,d exhibit higher diversity. From Figure 6a, we can see that the results have similar colors. Additionally, the buildings along the canal are very much alike. We can observe that the images in Figure 6b contain different types of buildings, terrains, and reflection patterns. It is clear that the proposed imagination model is able to generate images with larger variety. Although the generated image in the top right corner of Figure 6b is an unsuccessful case, such failed examples also happen in Figure 6c (image on row 3, column 2 and column 3) generated by AttnGAN. Both methods inevitably generate unsatisfying results occasionally.

For subjective qualitative evaluation, twenty users are invited to evaluate the results generated by the proposed method and AttnGAN. Each user is asked to evaluate the generated images using 30 different descriptions. For each description, twelve images are generated at a time so that the user can evaluate the variety of the generated images under the same description. Each user performs evaluation by choosing which result he or she prefers in terms of quality and variety. Among all user scores, 53% of them prefer the proposed method in terms of image quality. Additionally, 79% of the user scores prefer the proposed method in terms of variety. This subjective evaluation shows that the proposed method has both superior quality and more variety.

One possible application for the proposed framework is data augmentation for autonomous vehicles. Realistic and high-quality road scenes can be generated through the proposed system. For example, phrases such as “forest road in the morning”, “field road at night”, or “mountain road in winter” can be used to specify the kind of road scenes to be generated. With the help of additional image to video generation techniques proposed in [38,39,40], training and testing videos can be generated. This could save a lot of time and human power spent on driving and recording datasets under different weather conditions, in different seasons, and on different types of roads. Figure 7 displays the generated road images using the proposed framework. The descriptions used to generate the images from the first row to the last row in Figure 7 are “forest road in the morning,” “forest road in autumn,” and “forest road in winter.” The proposed system successfully gives the generated images desired characteristics described by the input texts. Additionally, the generated images exhibit sufficient variety to provide useful data augmentation for training so that the autonomous vehicles can adapt to different road conditions. From Figure 6 and Figure 7, we can see that the proposed framework is not limited to generating road images. It is also feasible to extend the framework to generate different types of scenes for miscellaneous robots or unmanned vehicles that can travel in different terrains or surfaces.

4. Conclusions

This paper proposes a framework to generate natural scenery images according to user descriptions. With text-to-image generation techniques, the proposed system can generate images under specified places, times and seasons. The framework improves and modifies the architecture of a generative adversarial network with attention models by adding the imagination models. The proposed imagination model utilizes the LSTM cell in the text encoder. It extracts the features again from the text descriptions in every up-sampling process. Such design enriches the text features given in the corpus description. We construct a Scene dataset which includes different types of scenery images for experimental purposes. The experiments show that the proposed framework can generate realistic and high-quality image scenes with sufficient diversity. The IS and FID scores are superior compared with the original AttnGAN. For future works, the work can be improved to generate more detailed descriptions specifying objects in the scenes. The work can also be extended to generate more general scene images in addition to natural sceneries. Moreover, the work can be combined with motion-generation techniques to generate videos. We can expect to find possible applications in data augmentation for autonomous vehicles by generating driving-training videos on desired type of roads and weather conditions with the help of the proposed framework.