High-Quality Text-to-Image Generation Using High-Detail Feature-Preserving Network

Abstract

Multistage text-to-image generation algorithms have shown remarkable success. However, the images produced often lack detail and suffer from feature loss. This is because these methods mainly focus on extracting features from images and text, using only conventional residual blocks for post-extraction feature processing. This results in the loss of features, greatly reducing the quality of the generated images and necessitating more resources for feature calculation, which will severely limit the use and application of optical devices such as cameras and smartphones. To address these issues, the novel High-Detail Feature-Preserving Network (HDFpNet) is proposed to effectively generate high-quality, near-realistic images from text descriptions. The initial text-to-image generation (iT2IG) module is used to generate initial feature maps to avoid feature loss. Next, the fast excitation-and-squeeze feature extraction (FESFE) module is proposed to recursively generate high-detail and feature-preserving images with lower computational costs through three steps: channel excitation (CE), fast feature extraction (FFE), and channel squeeze (CS). Finally, the channel attention (CA) mechanism further enriches the feature details. Compared with the state of the art, experimental results obtained on the CUB-Bird and MS-COCO datasets demonstrate that the proposed HDFpNet achieves better performance and visual presentation, especially regarding high-detail images and feature preservation.

1. Introduction

Generative adversarial networks (GAN) [1] and deep learning have recently been increasingly used in image and video generation and restoration [2,3,4,5,6,7,8] for the use and application of related optical devices. These networks are also commonly used to transform text descriptions into realistic images, known as the text-to-image task. A primary goal of artificial intelligence (AI) development is to enable AI machines to understand natural language and generate appropriate responses. To achieve this goal, various scholars have strived to improve the output quality of AI-generated images based on text prompts and ensure that these text-based images possess the detail required by users. Several methods have been proposed to improve the text-based image quality [9,10,11,12,13,14]. Stacked methods [9,11] were proposed to generate images through multiple stages and achieve improved image resolutions. Xu et al. [10] used an attention mechanism to combine text and image features, ensuring that the generated images matched the corresponding textual descriptions. Text-to-image and image-to-text models [13] were combined to improve the consistency between image details and textual descriptions. A dynamic memory network [14] was used to compute and integrate word-level features with image features. This method enabled the network to generate images that contained the details indicated by single words.

Although these methods have generated higher-quality images compared to previous approaches, several issues remain unresolved. Multistage methods [9,10,11,14] improve image resolutions and extract image and text features. They first extract text and image features from the previous stage and then implement a residual network [15] to retain relevant image features. A residual network uses skip connections to resolve the gradient vanishing problem commonly encountered in deep neural networks. In general, residual networks use nonlinear transformations to integrate the data of the current and previous stages, thereby preventing data loss. However, the number of parameters processed by the residual network increases in deeper network layers, causing the network to lose essential data when these data propagate through the bottleneck layer. Thus, information regarding the contextual representation of text and image fusion is lost, resulting in the calculated contextual representation being unusable for efficient image generation. Hence, producing the contextual representation requires additional computing resources.

To address these issues, the novel High-Detail Feature-Preserving Network (HDFpNet) is proposed to attempt to generate images with high detail and feature preservation. First, the initial text-to-image generation (iT2IG) module is used to generate initial feature maps. The fast excitation and squeeze feature extraction (FESFE) module is then proposed to recursively generate high-detail and feature-preserving images through channel excitation (CE), fast feature extraction (FFE), and channel squeeze (CS). Finally, the channel attention (CA) mechanism [16] further enriches the feature details. The experimental results indicate that the proposed HDFpNet achieves better performance and visual representation on the CUB-Bird [17] and MS-COCO [18] datasets than the state-of-the-art methods, especially regarding high-detail images and feature preservation.

We list some specific practical examples combined with our method and present schematic diagrams in Figure 1. By utilizing devices like mobile phones as sensors to capture text and analyzing it with the proposed HDFpNet model, we can enhance both understanding and safety. As shown in Figure 1a, when people see a warning sign but cannot recognize the language, they may face safety risks. At this point, by using a mobile phone or other device to sense the text and then analyzing it with the proposed HDFpNet model, relevant images can be generated to understand the meaning of the text, thereby avoiding dangerous behavior. In Figure 1b, we show another example in which people can use devices like mobile phones to sense operating instructions and obtain a process that matches the text, allowing individuals who do not understand the language to clearly comprehend the operational steps. Additionally, as shown in Figure 1c, when people see unfamiliar bird descriptions while birdwatching outdoors, they can use the text description and our method to generate images that match the text, making it easier to understand the appearance of the described bird. The examples above demonstrate the application of text descriptions combined with sensors, providing better visual assistance for machine operation and warning signs and offering a valuable technological application for those who have not personally encountered birds and other organisms. The contributions of this study are summarized as follows.

• The novel HDFpNet is proposed to be able to fully exploit a contextual representation of text and image fusion to generate high-quality, high-detail, and feature-preserving images efficiently in the use and application of optical devices.
• A novel FESFE module is proposed to preserve the contextual representation of text and image fusion through CE and CS blocks to avoid information loss before feature extraction, enabling the FFE block to extract features quickly without information loss.
• The experimental results indicate that the proposed HDFpNet achieves better performance and visual representation on the CUB-Bird and MS-COCO datasets than comparable state-of-the-art algorithms. The generated images are closer to the real ones and faithful to the text description, especially in terms of high image detail and feature preservation, making this method suitable for the use and application of optical devices.

2. Related Work

2.1. GANs

The effectiveness of GANs [1] and deep learning has been recently demonstrated by breakthroughs in various applications [2,3,4,5,6,7,8,9,10,11,12,13,14], including improving the accuracy of pedestrian detection by means of super-resolution [19], enhancing underwater images [20], and ensuring color constancy and color consistency [21]. Scholars have attempted to improve the efficiency of deep learning [22,23,24,25,26,27,28,29] and GANs [30,31,32], attaining favorable results; others have conducted image generation research based on these networks. Most GANs generate images using conditioning variables, including attributes or class labels [33,34,35]. GAN-based methods also use images as conditioning variables to generate or edit other images [36,37]. Text-to-image generators are a key focus in GAN research. Text-to-image generation aims to create near-realistic images based on a provided description in a natural language. To achieve this goal, the quality and content of the generated images must be in accordance with the provided descriptions.

2.2. Text-to-Image Generation Networks

Reed et al. [35] used GANs to convert text into images for text-to-image generation, developing a simple and effective GAN architecture and training strategy that can synthesize captivating bird and flower images from human-written descriptions. However, the resolution of these images is very low. To address this issue, Zhang et al. proposed using StackGAN [9] and StackGAN++ [11], two multistage methods in which the image generated at each stage is used as input for the next stage of image generation, thereby increasing the resolution of the image. However, the generated images still cannot reflect the shapes or colors of objects and generate unreasonable bird images. AttnGAN [10] introduced an attention mechanism into multistage image generation, allowing the model to perform word-level processing and enhance the image details while effectively generating text-to-image content for complex scenes. However, there are still many distorted images generated, such as birds with two heads, which are illogical.

DM-GAN [14] introduced a dynamic memory generation network (DM-GN) to adjust the weight of each word in a sentence and enhance the blurry parts of the image generated in the previous stage. By assigning weights reflecting the importance of text individually, it improves the quality of the final image. However, it emphasizes the overall completeness and clarity of the image generated after text generation, while ignoring illogical local details. MirrorGAN [13] reconstructed text descriptions during image generation to improve the consistency between text and images. Since it emphasizes the consistency between the generated images and text descriptions, it ignores the effect of details, including the restoration of eyes and feet. KT-GAN [12] used a knowledge transfer method from image-to-image models to support text-to-image models, thereby improving the image quality. However, the generated images do not restore the features well. The recently proposed Transformer-based text-to-image methods can generate more complex images by calculating a large number of text parameters and combining them with an image encoder. CogView [38] and DALL-E [39] leveraged the advantages of Transformer-based methods to generate visually reasonable images in various types of scene descriptions. Although they can generate rough background outlines, the former produces images with relatively blurry backgrounds, while the latter has insufficient restoration. CSM-GAN [40] proposed a cross-modal semantic matching method that can better encode local key information to highlight detailed features in synthesized images and highlight important local structural information in text descriptions. However, it ignores the details in the picture, such as a bird’s feet, which may be presented in an illogical manner, such as having only one foot or abnormally fine feet. DR-GAN [41] generated images from text descriptions through improved distribution learning, obtaining more accurate image distribution from key information to generate better-quality images. Although the overall performance regarding details is better, there are still some problems, including some artifacts and details, such as birds’ feet not being well presented.

The multistage methods [9,10,11,12,13,14] primarily aim to enhance the image resolution and improve the extraction of image and text features. The extracted features are processed using a residual network to retain relevant features. However, none of these methods optimize the current image features. As the depth of the network layer increases, the number of parameters processed by the residual network also increases; some data may be lost during propagation through the bottleneck layer. To address these issues, the HDFpNet model is proposed to retain the image features and improve the image quality in this study.

3. Method

The network architecture of the proposed HDFpNet is shown in Figure 2. It consists of three components: the iT2IG module, the FESFE module, and the CA mechanism.

3.1. iT2IG Module

The transformation of text descriptions to image feature maps is a vital topic in text-to-image generation because the generation of an adequate feature map is important for subsequent image enhancement. The first component of the proposed framework is the iT2IG module (Figure 2a), which performs initial feature generation. The module incorporates a text encoder, an initial image generator G₀, and a dynamic memory generative network (DM-GN) [14].

• Text Encoder: Following the methods adopted in [10,14], textual data are processed by using a pre-trained bidirectional long short-term memory (LSTM) text encoder [10] to transform descriptive sentences into sentence features (s) and word features (W).
• Initial Image Generator G₀: The conditioning augmentation $F^{ca}$ proposed in [4] is used to optimize sentence features. Next, G₀ is used to generate the initial image $I_0$ and image features $H_0$,

$$z: I_0,H_0=G_0\left(z, F^{ca}\left(s\right)\right),$$

(1)

where $z ~ N(0,1)$, and G₀ denotes the initial image generator. This generator is composed of four upsampling blocks. $F^{ca}$ denotes the conditioning augmentation block [11].

3.

DM-GN: The DM-GN [14] is used to generate feature maps because its effective image enhancement method dynamically adjusts word weights to achieve images with more word-level detail. As presented in Figure 2a, the inputs of the DM-GN are the initial image features H_i and the word features W to the DM-GN:

$$\mathrm{W}=\{w_1,w_2, w_3,\dots ,w_T\}, w_i\in {\mathbb{R}}^{N_w}$$

(2)

$$H_i=\{h_1,h_2,h_3,\dots ,h_N\}, h_i\in {\mathbb{R}}^{N_r}$$

(3)

where T denotes the number of words, $N_w$ denotes the dimensions of word features, N denotes the number of pixels, and $N_r$ denotes the dimensions of the image features. In this study, $N_w=256, N_r=64$. The image feature and word feature vectors are input into the DM-GN to perform feature recombination. To process text features, 1 × 1 kernels are used to select word features, which are then placed in a feature space,

$$m_i=Conv\left(w_i\right), m_i\in {\mathbb{R}}^{N_m},$$

(4)

where $Conv\left(\right)$ denotes the selection of word features by the 1 × 1 kernels, $N_m$ denotes the dimensions of the feature space ($N_m=128$), and m denotes the feature space. Equation (4) is not used to process the corresponding word features to select specific word vectors and extract image features. Instead, the word features W are integrated with the image features $H_i$ generated in the previous stage to determine the weight of each word. The weight is calculated as

$$d_i^w(H,w_i)=\delta (A\ast w_i+B\ast {\displaystyle \frac{1}{N}}\sum _{i=1}^N{h}_i),$$

(5)

where $\delta \left(\right)$ denotes the sigmoid activation function, A denotes a 1 × $N_w$ matrix, and B denotes a 1 × $N_r$ matrix. After the weight of each word is calculated, the word and image features are recalculated based on the corresponding weights and placed in the feature space $m_i\in R^{N_m}$,

$$m_i=C_w\left(w_i\right)\ast d_i^w+C_r({\displaystyle \frac{1}{N}}\sum _{i=1}^N{h}_i)\ast \left(1-d_i^w\right), m_i\in R^{N_m},$$

(6)

where $C_w$ and $C_r$ denote the selection of word and image features, respectively, by the 1 × 1 kernels, and m denotes the feature space with dimension $N_m$. After the image and word features are integrated, the weight of the jth image feature in the ith feature space is calculated:

$${\alpha }_{i,j}={\displaystyle \frac{\exp \left({\varnothing }_K\right(m_i{)}^T{h}_j)}{\sum _{l=1}^T\exp \left({\varnothing }_K\right(m_l{)}^T{h}_j)}},$$

(7)

where ${\alpha }_{i,j}$ denotes the similarity probability between the ith feature space and jth image feature, and ${\varnothing }_K$ denotes the feature selection process by the 1 × 1 kernels. The dimension of the feature space is changed to Nr to ensure that the image space and image feature space have the same dimensions. After the weight calculation, ${\alpha }_{i,j}$ and the feature space $m_i$ are used:

$$o_j=\sum _{i=1}^T{\alpha }_{i,j}\varnothing V(m_i),$$

(8)

where o denotes the weighted feature space, and $\varnothing V$ denotes the feature selection process by the 1 × 1 kernels. The dimension of the feature space is changed to Nr to ensure that the image space and image feature have the same dimensions. o is integrated with the corresponding image features to create a set of initial image features $h$,

$$h_i=\left[o_i,h_i\right],$$

(9)

where [·,·] denotes a series operation that matches each image feature to its corresponding textual description.

3.2. FESFE Module

After the weight calculation, the image enhancement block must retain relevant image features in the next stage. In this study, a novel FESFE module is proposed to address the shortcomings of conventional residual blocks by retaining image features. This module contains three blocks: CE, FFE, and CS. These blocks are illustrated in Figure 2b–d, respectively.

• CE Block: Based on previous studies on deep residual networks [35,42,43], the present study introduces a receptive field to improve the relevance of the prediction models and increase the number of extracted image details. Previous studies have used 3 × 3 filters without dilation to extract features [10,14]. The proposed CE block optimizes and excites existing channels to increase the effectiveness of subsequent feature extraction (Figure 2b). The purpose of the CE block is to create a channel excitation effect that increases the number of channels, enabling a subsequent feature extraction process without feature loss. Batch normalization (BN) [37] is also implemented in the excitation process to optimize the overall network; $BN(x)={\displaystyle \frac{h-mean\left(x\right)}{\sqrt{Var\left(x\right)}+eps}}$. Each iteration of BN might result in a small shift, which causes considerable changes to the network output. Therefore, a swish activation function [37], $\mathrm{f}(x)={\displaystyle \frac{x}{1+e^{-x}}}$, is added to improve the BN process. The proposed method enables the extraction of more image details than in existing methods, thereby generating images that more closely resemble the target images. Mathematically, the process can be described as follows:

$$p_i=\rho \left(BN\left(C_{FE}\left(h_i\right)\right)\right), p\in {\mathbb{R}}^{{2N}_m},$$

(10)

where $C_{FE}$ denotes a filter composed of a 1 × 1 kernel; the filter doubles the number of channels.$BN\left(\right)$ denotes the BN process [37], and $\rho \left(\right)$ denotes the swish activation function [38].

2.

FFE Block: Extending the dimensions of the CE block (Figure 2b) increases the computational cost. To accelerate the feature extraction process without reducing the model’s effectiveness, an FFE block (Figure 2c) is incorporated into the proposed model based on previous research [42,43]. The FFE block performs convolutions on the spatial and depth dimensions (i.e., channel dimensions) to reduce the computations and quickly extract features. The block includes both depth-wise convolution and point-wise convolution calculations.

$$p_i^{DW}=C_{DW}\left(p_i\right), p\in {\mathbb{R}}^{{2N}_m},$$

(11)

$$p_i^{PW}=C_{PW}\left(p_i^{DW}\right), p\in {\mathbb{R}}^{{2N}_m},$$

(12)

where $C_{DW}\left(\right)$ denotes a depth-wise convolution with N 5 × 5 kernels, where N is the number of channels, to calculate and stack the channels in each layer. $C_{DW}\left(\right)$ denotes point-wise convolution, in which 1 × 1 kernels are used to generate a new feature map.

3.

CS Block: After a new feature map is obtained, the CS block performs a channel squeeze process to accelerate the data propagation speed (Figure 2d) and reduce the number of channels to its original value, namely that obtained after the iT2IG module calculation.

$$p^{FC}=\rho \left(BN\left(C_{FC}\left(p_i\right)\right)\right), p\in {\mathbb{R}}^{N_m},$$

(13)

where $C_{FC}$ denotes the use of 1 × 1 filters to reduce the number of channels to their original value. $BN\left(\right)$ denotes the BN process [37], and $\rho \left(\right)$ denotes the swish activation function [38].

3.3. CA Mechanism

Finally, squeeze-and-excitation (SE) networks [16] are used to improve the quality of the feature map. These networks can learn an information relationship between channels to calculate the weights for feature adjustment in each channel. Figure 2e presents the architecture of the squeeze-and-excitation-based CA mechanism. This mechanism recalibrates the features in an input feature map through the “squeeze” process, in which each input feature map undergoes global average pooling to compress the two-dimensional feature maps of all channels into a single global feature map comprising every feature in each map. The output of global average pooling is calculated as follows:

$$z_c=F_{sq}\left(p^{FC}\right)={\displaystyle \frac{1}{H\times w}}\sum _{i=1}^H\sum _{j=1}^W{p}^{FC}(i,j),$$

(14)

where $p^{FC} \in R^{H\times W}$ denotes the input feature maps, and $z_c$ denotes the compression result. After the channel information from the squeeze process is obtained, the “excitation” process is performed to predict the importance of each channel. This is achieved using two fully connected layers, a rectified linear activation function, and a sigmoid function. The channel weights and feature mapping weights obtained from model learning are applied to the generated channel information, which is the input features. The output of the SE process is calculated as

$$s=F_{se}\left(Z_c,W\right)=\delta \left(W_2\sigma \left(W_1{Z}_c\right)\right),$$

(15)

where $W_1$ and $W_2$ denote the parameters of the first and second fully connected (squeeze) layers, respectively; $\delta \left(\right)$ denotes the sigmoid function; and $\sigma \left(\right)$ denotes the rectified linear activation function. Finally, the output weights of the excitation operation are multiplied by the original features to further enhance the essential features, which are used as the final output of the SE module.

$${\tilde{x}}_c=F_{scale}\left(p^{FC},s_c\right)=s_c·p^{FC},$$

(16)

Finally, the output of the HDFpNet model is calculated as follows:

$$h^{NEW}=h+{\lambda }_h\times F_{\mathrm{H}\mathrm{D}\mathrm{F}\mathrm{p}\mathrm{N}\mathrm{e}\mathrm{t}}\left(h\right),$$

(17)

where ${\lambda }_h$ denotes a hyperparameter that is equal to 0.1. $F_{\mathrm{H}\mathrm{D}\mathrm{F}\mathrm{p}\mathrm{N}\mathrm{e}\mathrm{t}}\left(\right)$ denotes the calculation process of the FESFE module and CA mechanism. This process is repeated three times in each stage.

3.4. Loss Functions

In this study, GAN loss functions are combined with other loss functions to calculate the match level between text and image features. The image generator is trained with the loss function $L_{G_i}$, which is expressed as follows:

$$L_{G_i}=-{\displaystyle \frac{1}{2}}\underset{\mathrm{unconditional} }{\underbrace{[E_{x\sim P{G}_i}\mathit{log}{D}_i(x)}}+\underset{\mathrm{conditional}\mathrm{loss} }{\underbrace{E_{x\sim P{G}_i}{D}_i(x,s)}}],$$

(18)

where the first term represents unconditional losses, enabling the generation of images that closely resemble real images. The second term represents conditional losses, ensuring that the generated image’s content matches the description of the input sentences.

The discriminator is trained with the loss function $L_{D_i}$, which is expressed as follows:

$$\begin{array}{cc}\hfill L_{D_i}=& -\underset{\mathrm{unconditional}\mathrm{loss}}{\underbrace{{\displaystyle \frac{1}{2}}[E_{x\sim \mathit{Pdata}}\mathit{log}{D}_i(x)+E_{x\sim P{G}_i}\log (1-D_i(x))}},\hfill \\ & +\underset{\mathrm{conditional} \mathrm{loss}}{\underbrace{E_{x\sim \mathit{Pdata}}\mathit{log}{D}_i(x,s)+E_{x\sim P{G}_i}\log (1-D_i(x,s\left)\right)}}],\hfill \end{array}$$

(19)

where the unconditional loss is designed to distinguish the generated image from real images, and the conditional loss determines whether the image and the input sentence match. Text processing using the CA loss function was proposed in [9]. This function uses an independent Gaussian distribution to resample sentence vectors and prevent overfitting, thereby increasing the volume of training data. The CA loss function $L_{CA}$ is expressed as follows:

$$L_{CA}=D_{KL}\left(\mathrm{N}\right(\mathsf{\mu }\left(\mathrm{s}\right),\mathsf{\Sigma }\left(\mathrm{s}\right)\left)\right|\left|\mathrm{N}\left(0,\mathrm{I}\right)\right),$$

(20)

where $\mathsf{\mu }\left(\mathrm{s}\right)$ denotes the mean sentence vector and $\mathsf{\Sigma }\left(\mathrm{s}\right)$ denotes a covariance matrix. The values of $\mathsf{\mu }\left(\mathrm{s}\right)$ and $\mathsf{\Sigma }\left(\mathrm{s}\right)$ are calculated by fully connected layers. The deep attentional multimodal similarity model [3] determines the match level between the word and image features. Accordingly, the total loss function L is expressed as follows:

$$L=L_{G_i}+{\lambda }_1{L}_{CA}+{\lambda }_2{L}_{DAMSM}.$$

(21)

where ${\lambda }_1$ and ${\lambda }_2$ are hyperparameters.

4. Experiments

4.1. Experimental Settings

• Implementation Details: To generate an initial feature map, text processing was performed using a bidirectional LSTM text encoder as proposed in [10]. This encoder transformed each textual description into two hidden states representing the forward and backward sentences. Each word also had two corresponding hidden states. These two hidden states for each sentence were concatenated and then input into the initial image generator to create the initial image features. Three image generation stages were conducted to produce images with resolutions of 64 × 64, 128 × 128, and 256 × 256 in sequence. Subsequently, the dynamic network generator proposed in [14] was employed to integrate the hidden states of each word with the initial image features. By employing the training method described in [14], the CUB-Bird dataset parameters were configured as ${\mathsf{\lambda }}_1=1$ and ${\mathsf{\lambda }}_2$ = 5, while the MS-COCO dataset parameters were configured as ${\mathsf{\lambda }}_1=1$ and ${\mathsf{\lambda }}_2$ = 50. After generating the initial text–image feature map, 1 × 1 filters doubled the number of feature map channels. Then, depth-wise and point-wise convolution were performed using 5 × 5 and 1 × 1 filters, respectively. Finally, 1 × 1 filters were employed to reduce the number of feature map channels to its original value (${\mathrm{N}}_{\mathrm{m}}$= 128). All networks were trained using the Adam optimizer [39] with a batch size of 16 and a learning rate of 0.0002. The CUB-Bird and MS-COCO datasets were used to train the HDFpNet model for 600 and 200 epochs, respectively. The model training process was conducted using an NVIDIA RTX A5000 card.
• Dataset: Two public datasets were utilized in this study—the CUB-Bird dataset and the MS-COCO dataset. The CUB-Bird dataset comprises 11,788 images of 200 bird species, with each image being accompanied by ten descriptive sentences. The MS-COCO dataset contains 80,000 training images and 40,000 test images, with each image being associated with five textual annotations.
• Evaluation: The performance of the HDFpNet model was evaluated through a series of experiments. Initially, the effectiveness of each block within the model was assessed. Subsequently, the performance of the model was compared with that of other state-of-the-art text-to-image generation models. For a task involving the generation of 30,000 images from textual descriptions recorded in a dataset that had not been used for testing other models, the Fréchet inception distance (FID) [42] and R-precision indicators were employed to evaluate the model’s performance as direct comparisons were not possible. The FID is based on features extracted from the Inception v3 network [44] and is used to calculate the Fréchet distance between generated and real-world images. A low FID value indicates that a generated image closely resembles a real-world image. R-precision evaluates the extent to which a generated image matches the conditions specified in a textual description. Specifically, it employs a query mechanism to determine the relevance of documents retrieved by a system. This study implemented R-precision following the procedures described in [10]. The cosine distance between the global image vector and 100 candidate sentence vectors was calculated. The candidate documents described R real images and 100 randomly selected unmatched samples. In each query, if r results obtained from the first R retrieved documents are relevant, then the R-precision equals r/R. In this study, R was set to 1. The generated images were divided into 10 leaflets for the performance of queries. The obtained scores are presented as means with standard deviations.

4.2. Comparisons with the State-of-the-Art Algorithms

To validate the performance of the proposed method, HDFpNet was compared with state-of-the-art approaches on the CUB-Bird and MS-COCO datasets. The FID and R-precision results are presented in

Table 1. Quantitative evaluation of different methods (StackGAN [9], AttnGan [10], ControlGAN [40], MirrorGan [13], KT-GAN [12], DM-GAN [14], TIME [14], and HDFpNet) in terms of FID↓ on CUB-Bird and MS-COCO datasets. Lower is better for FID.

Dataset	StackGAN [9]	AttnGAN [10]	ControlGAN [40]	MirrorGAN [13]	KT-GAN [12]	DM-GAN [14]	TIME [14]	Ours
CUB	--	23.98	--	--	17.32	16.09	14.3	13.89
COCO	74.05	81.59	35.49	--	30.73	32.64	31.14	27.21

and

Table 2. Quantitative evaluation of different methods (StackGAN [9], AttnGAN [10], ControlGAN [40], MirrorGAN [13], KT-GAN [12], DM-GAN [14], TIME [14], and HDFpNet) in terms of R-precision↑ on CUB-Bird and MS-COCO datasets. Higher is better for R-precision.

Dataset	StackGAN [9]	AttnGAN [10]	ControlGAN [40]	MirrorGAN [13]	KT-GAN [12]	DM-GAN [14]	TIME [14]	Ours
CUB	10.37	67.82	69.33	69.58	--	72.31	71.57	84.33
COCO	--	83.53	82.43	84.21	--	91.87	89.57	92.44

, respectively. As shown in Table 1, HDFpNet outperformed the compared methods in terms of the FID, with lower FID scores being indicative of better performance. For example, on the CUB-Bird dataset, HDFpNet and DM-GAN achieved FID scores of 13.89 and 16.09, respectively, while, on the MS-COCO dataset, their respective FID scores were 27.21 and 32.64. The lower FID scores achieved by HDFpNet demonstrate that it obtains a superior data distribution. Table 2 also reveals that the proposed HDFpNet model achieved R-precision scores of 84.33% and 92.44% on the CUB-Bird and MS-COCO datasets, respectively. These scores are notably higher than those achieved by the other methods. The high R-precision scores indicate that the proposed model generated images that closely matched their corresponding descriptions, thereby verifying its effectiveness. Figure 3 presents examples of images generated by HDFpNet and the state-of-the-art models. Overall, the HDFpNet model generated images with greater detail than the StackGAN [9], AttnGAN [10], and DM-GAN [14] models. This superior performance can be attributed to the fact that HDFpNet retained the image features selected by DM-GAN, enabling it to better understand the logic of the input description and present a clearer image structure. An analysis of the image samples generated from the CUB-Bird dataset (Figure 3) indicates that, at the word level, the AttnGAN [10] and DM-GAN [14] models performed well in capturing and presenting text features. However, HDFpNet was even more effective in displaying word-level features, generating images that accurately reproduced object features and structures in detail. These images closely matched the corresponding input text and contained fine details such as the feather textures and other anatomical characteristics (e.g., eyes and legs) of birds (Figure 3). As a result, the generated bird images closely resembled real images. These findings confirm that the FESFE module enables the HDFpNet model to generate more realistic images. Furthermore, the MS-COCO dataset was used to test the network’s ability to generate images containing multiple objects (Figure 4). When an input text described multiple themes or objects with complex structures, generating an accurate image was challenging. In each image, HDFpNet could identify the most essential themes and capture the main scene, with the remaining textual content being sorted in a logical sequence to arrange the objects in the image. For instance, as shown in the second row of Figure 4, HDFpNet could accurately generate an image of a room with wooden textural characteristics. These results further verify that the proposed model can enhance image details and follow textual logic.

Figure 5 presents comparisons of real images with those generated by the HDFpNet model at various image enhancement stages. The generated images were based on the feature maps created by the iT2IG module using the CUB-Bird dataset. The results reveal that, even before the final enhancement stage, the generated images already contained all of the required details. The final enhancement stage further improved these details to create more realistic images. The feature maps enhanced by the FFE block and CA mechanism facilitated the generation of objects with accurate structures and colors. The feature enhancement process at each stage involved optimizing the feature map generated in the previous stage. This process significantly improved the object structures and details, such as eyes and texture features, resulting in generated images that closely resembled real images. Figure 6 presents various types of images generated by HDFpNet using the same text descriptions from the CUB-Bird dataset. Despite being generated from the same input text, these images exhibit diverse styles and satisfactory quality in terms of object structures and textural details.

4.3. Ablation Study

(1)

Effectiveness of FESFE Module and CA Mechanism: The proposed HDFpNet model comprises the iT2IG module, which is based on DM-GAN [14]; the FESFE module; and the CA mechanism. An ablation experiment was conducted to evaluate the effectiveness of the CE block and CS block, the FFE block in the FESFE module, and the CA mechanism. The results are presented in

Table 3. Performance evaluation of HDFpNet in different model combinations on the CUB-Bird dataset in terms of the FID and R-precision. Baseline represents the generation of feature maps only using the It2IG module. CE, CS, FEE, and CA represent the CE block, CS block, FFE block, and CA mechanism, respectively.

Architecture	FID ↓	R-Precision ↑
Baseline	22.94	71.40%
+CE +CS	21.45	79.12%
+CE +FFE +CS	15.51	82.57%
+CE +FFE +CA +CS	17.50	81.38%
+CE +FFE +CS +CA	13.89	84.33%

. The feature extraction capability of the proposed model was significantly improved by increasing the number of channels, while including the CA mechanism optimized the overall network structure. The results demonstrate that the combined use of these blocks enabled the outstanding performance of the proposed model for text-to-image generation, thereby verifying the effectiveness of the FESFE module and CA mechanism. Notably, the +CE +FFE +CA +CS sample in the fourth row of Table 3 was created by extracting features through the FFE block before they were processed by the CA mechanism and then by the CS block. This sample was created to verify the effectiveness of the CA mechanism in the HDFpNet model. The results confirm that initiating the CA mechanism while the FFE block was still extracting features reduced the model’s performance. In contrast, initiating the CA mechanism after feature extraction by the FFE block had been completed yielded optimal model performance.

(2)

Study of ${\lambda }_h$: The hyperparameter ${\lambda }_h$ was employed to improve the performance of the HDFpNet model. An experiment was conducted using two ${\lambda }_h$ values: 0.1 and 1. The larger ${\lambda }_h$ value (i.e., ${\lambda }_h$ = 1) was found to have a negative impact on the model performance (

Table 4. Performance evaluation of different ${\mathsf{\lambda }}_h$ on CUB-Bird dataset in terms of FID and R-precision.

${\mathsf{\lambda }}_{\mathit{h}}$	FID ↓	R-Precision ↑
${\mathsf{\lambda }}_h=0.1$	13.89	84.33%
${\mathsf{\lambda }}_h=1.0$	15.42	82.91%

). As a result, ${\lambda }_h$ = 0.1 was used for subsequent analyses. Evaluations based on the FID and R-precision indicators also confirmed that this value yielded the optimal results.

(3)

Study of ${\mathrm{F}}_{\mathrm{H}\mathrm{D}\mathrm{F}\mathrm{p}\mathrm{N}\mathrm{e}\mathrm{t}}$: An experiment was conducted to determine the optimal number of $F_{\mathrm{H}\mathrm{D}\mathrm{F}\mathrm{p}\mathrm{N}\mathrm{e}\mathrm{t}}$ iterations required to generate the best feature map. Between one and five iterations were performed. It was found that lower iteration numbers (i.e., ≤3) resulted in higher performance, with the optimal performance being achieved at three iterations (

Table 5. Performance evaluation of HDFpNet in different iterations in terms of FID and R-precision.

Iteration (Times)	FID ↓	R-Precision ↑
1	25.75	81.12%
2	14.301	82.72%
3	13.89	84.33%
4	14.91	83.15%
5	16.1153	83.25%

). In contrast, iteration numbers > 3 reduced the model performance. As a result, the iteration number was set to three to obtain optimal results in the subsequent analyses. This configuration was verified to yield outstanding image generation and quality results.

5. Conclusions

The novel HDFpNet network has been proposed for high-detail and feature-preserving text-to-image generation tasks and requires fewer feature computation resources. The iT2IG module is first used to generate initial feature maps to avoid the loss of initial image details and reduce feature loss. The FESFE module is proposed to efficiently and recursively generate high-detail and feature-preserving images using CE, FFE, and CS blocks with a lower feature computation cost. It can significantly avoid feature loss and simultaneously enhance the image details to greatly improve the quality of the generated images. Finally, the CA mechanism is adopted to further enrich the feature details. The experimental results indicate that the proposed HDFpNet has more promising performance and better visual presentation than the state-of-the-art approaches on the CUB-Bird and MS-COCO datasets, especially in terms of high-detail images and feature preservation. In future work, studies can be expanded to various applications of optical devices, such as creating art using cameras and smartphones, translating text into images for web applications using smartphones and other optical devices, and creating data visualizations with smartphones.