IAACLIP: Image Aesthetics Assessment via CLIP

Abstract

Aesthetics primarily focuses on the study of art, encompassing the aesthetic categories of beauty and ugliness, as well as human aesthetic activities. Image Aesthetics Assessment (IAA) seeks to automatically evaluate the aesthetic quality of images by mimicking the perceptual mechanisms of humans. Recently, researchers have increasingly explored using user comments to assist in IAA tasks. However, human aesthetics are subjective, and individuals may have varying preferences for the same image, leading to diverse comments that can influence model decisions. Moreover, in practical scenarios, user comments are often unavailable. Thus, this paper proposes a CLIP-based method for IAA (IAACLIP) using generative descriptions and prompts. First, leveraging the growing interest in multimodal large language models (MLLMs), we generate objective and consistent aesthetic descriptions (GADs) for images. Second, based on aesthetic images, labels, and GADs, we introduce a unified contrast pre-training approach to transition the network from the general domain to the aesthetic domain. Lastly, we employ prompt templates for perceptual training to address the lack of real-world comments. Experimental validation on three mainstream IAA datasets demonstrates the effectiveness of our proposed method.

1. Introduction

Aesthetics is a discipline primarily focuses on the study of art, exploring the aesthetic categories of beauty and ugliness, as well as human aesthetic activities. Image Aesthetics Assessment (IAA) [1,2,3] seeks to automatically evaluate the aesthetic quality of images by mimicking the perceptual mechanisms of humans. IAA is categorized into two types: General Image Aesthetics Assessment (GIAA) and Personalized Image Aesthetics Assessment (PIAA). GIAA pertains to the collective perception of image aesthetics shared by most people, whereas PIAA [4,5,6,7] focuses on the distinct aesthetic views of individual users. Benefiting from the recent advancements in science and technology, IAA applications have expanded to fields like style transfer [8], image retrieval [9], and image enhancement [10]. However, due to its inherent subjectivity, it still faces challenges.

In the early stages of IAA research [11,12,13,14], researchers constructed handcrafted features based on composition rules and photography standards. This approach has some drawbacks, as the complex extraction of handcrafted features incurs significant costs and cannot fully capture the subtle differences in human aesthetic perception of various photographic principles. Driven by the rapid advancements in deep learning [15,16,17,18,19,20,21,22,23], IAA models are generally developed to tackle three key tasks: the binary classification of aesthetics [11], regression of aesthetic scores [16], and prediction of aesthetic distributions [17]. However, current methods mostly focus on the visual aspect. Celona et al. [20] developed a method for automatically evaluating the aesthetic appeal of images by analyzing their semantic content, artistic style, and composition.

Previous approaches have predominantly assessed aesthetic quality based solely on visual data, which presents inherent limitations. Several studies have also explored the combination of visual and textual information for IAA. Nie et al. [22] aimed to simulate human implicit memory to trigger subjective aesthetic expression, proposing a brain-inspired multimodal interaction network to accomplish IAA tasks by mimicking how associative areas of the brain cortex process sensory stimuli. Some multimodal methods usually rely on both images and comments as input during testing, which restricts their practical applicability in real-world scenarios. A promising approach, as described in [23], involves using multimodal IAA datasets for model training, enabling the trained image encoder to directly conduct aesthetic inference during testing. Another promising approach is to employ visual–textual bimodality during training and to construct prompt templates during testing to assist IAA tasks.

CLIP [24] is an artificial intelligence model with multimodal understanding capabilities. In [23,25], the researchers noted that due to CLIP’s extensive pre-training on diverse data, it proves highly effective in multimodal tasks. However, applying CLIP to the field of IAA presents challenges. Therefore, there is a need to devise a method to transition CLIP from the general domain to the aesthetics domain. Previous studies have utilized images from aesthetic databases along with corresponding user comments. However, we found that human aesthetic preferences are diverse, and different individuals offer varying evaluations of the same image. Using these comments may interfere with the model’s aesthetic evaluation, as illustrated in Figure 1. This figure illustrates the text descriptions in the general domain and the image comments in the aesthetic domain. It is evident that there is a domain gap between the two. For example, in the user comments in the aesthetic database, some individuals may describe the image as “very beautiful”, indicating a positive evaluation, while others may feel that the image lacks focus and fails to attract their attention. These are two entirely different types of comments. Some existing multimodal aesthetic evaluation methods use the original user comments, but we consider this approach somewhat problematic. With the rise of multimodal large language models (MLLMs), we employ the MLLM ChatGPT4 to generate objective and consistent comments, thereby facilitating the transition from the general to the aesthetic domain.

Building on these considerations, we propose a CLIP-based framework comprising two key stages: unified contrastive pre-training and perception training. In the unified contrastive pre-training stage, we utilize generated aesthetic descriptions to construct aesthetic image–description pairs. Through this process, CLIP is adapted to the aesthetic domain by learning to associate images with their corresponding aesthetic descriptors. In the perception training stage, we design structured prompt templates based on the generated aesthetic descriptions. These templates, combined with images, are then employed for IAA tasks. This approach effectively addresses the scarcity of real-world aesthetic image comments while providing a flexible and customizable framework for aesthetic evaluation.

The main contributions of this work are as follows:

1.

Unlike other multimodal methods, we utilize MLLMs to generate objective and consistent aesthetic descriptions for aesthetic images to assist in IAA tasks.

2.

Based on the generated aesthetic image–description pairs, we propose the use of unified learning methods to complete the unified contrastive pre-training stage of our proposed method, facilitating CLIP’s transition from the general to the aesthetics.

3.

We construct composite prompt templates based on the generated aesthetic descriptions to realize the perception training stage of our proposed method. This approach addresses the lack of comments in the real world, and according to experiments conducted on three IAA databases, our method exhibits a competitive performance compared to other state-of-the-art (SOTA) models.

The remainder of this article is organized as follows. Related works are briefly reviewed in Section 2. Section 3 introduces the details of IAACLIP. Section 4 shows the experimental results. Finally, the conclusions are drawn in Section 5.

2. Related Works

2.1. Image Aesthetics Assessment

General Image Aesthetics Assessment (GIAA): Early GIAA work focused on using handcrafted features to describe image aesthetics. Datta et al. [11] established a significant correlation between various visual characteristics of photographic images and their aesthetic ratings. By employing an SVM classifier, they achieved good accuracy in distinguishing high-rated and low-rated photos using 15 visual features. Ke et al. [12] proposed a principled approach to designing advanced features for photo quality assessment, considering multiple perceptual features to make aesthetic decisions.

In past research, handcrafted features achieved limited success in the IAA field because the nature of aesthetics is highly abstract, making it difficult for handcrafted features to fully capture it. Benefits to the advancements in deep learning include the approach to handling IAA shifting from handcrafted features to deep features [26,27,28,29,30,31]. Lu et al. [26] merged global and local views generated from images and used a dual-column deep convolutional neural network for IAA tasks. Ma et al. [28] proposed an architecture capable of accepting images of arbitrary sizes and learning aesthetic knowledge from both details and overall image layouts simultaneously. Huang et al. [29] proposed a coarse-to-fine image aesthetic evaluation model guided by dynamic attribute selection, simulating human aesthetic perception processes by conducting coarse-to-fine aesthetic reasoning.

Personalized Image Aesthetics Assessment (PIAA): PIAA is subjective and individual, reflecting personal preferences and emotional responses to images. Ren et al. [4] proposed two personalized aesthetics datasets to predict individuals’ opinions on general image aesthetics. Since PIAA is a typical few-shot task, ref. [5,6] proposed to introduce meta-learning to solve this problem. Zhu et al. [5] introduced a bi-level optimization meta-learning algorithm to solve the PIAA task. Yan et al. [6] proposed a dual-column network based on meta-learning by considering the global and local features in personalized aesthetic images. In [7], a new user-guided PIAA framework was introduced, which leverages user interactions to adjust and rank images, using deep reinforcement learning (DRL) for aesthetic evaluation.

2.2. Multimodal Learning in IAA

Multimodal approaches have shown significant improvements in performance compared to using single modals alone. In recent IAA studies [22,23,32], researchers have applied multimodal learning to the field with great success. Sheng et al. [23] introduced a CLIP-based IAA framework using multi-attribute contrastive learning, with an emphasis on fine-grained aesthetic domain attributes in comments. They categorized comments into different classes to exploit the rich semantics of these attributes and proposed attribute-aware learning to capture the relationship between images and their aesthetic features. Nie et al. [22] argued that existing multimodal IAA approaches overlook the interactions between modalities. They proposed a brain-inspired multimodal interaction network, simulating how associative areas of the brain cortex process sensory stimuli. They introduced Knowledge-Integrated LSTM to learn the interaction between images and text. The Scalable Multimodal Fusion approach, based on low-rank decomposition, integrates image, text, and interaction modalities to predict the final aesthetic distribution.

2.3. Multimodal Large Language Models

In recent years, driven by improvements in hardware capabilities and the progress of deep learning technologies, large language models (LLMs) have made remarkable progress. Models like ChatGPT and LLaMA [33] have achieved significant milestones in understanding, reasoning, and generation, drawing considerable attention from both academia and industry. Building on the achievements of LLMs in natural language processing, the rise in multimodal large language models (MLLMs) has ignited a fresh wave of innovation in visual language processing. Notable works such as MiniGPT [34], Otter [35], and LLaVA [36] have demonstrated exceptional capabilities in combining visual and textual information. However, the performance of MLLMs in image aesthetic perception remains uncertain. To address this, Huang et al. [37] proposed an expert benchmark called AesBench. First, an expert-labeled aesthetic perception database (EAPD) was constructed, featuring diverse image content and high-quality annotations. Second, a set of comprehensive criteria was proposed to assess the aesthetic perception abilities of MLLMs from four dimensions. Based on the PARA dataset, we construct aesthetic descriptions using the MLLM ChatGPT4.

3. Methodology

3.1. Overview

The model presented in this work is built upon the pre-trained CLIP model, which features two encoder branches for processing text and images simultaneously. Our approach involves two training stages, as illustrated in Figure 2: the unified contrastive pre-training stage and the perceptual training stage. To address the cross-domain challenges in CLIP, we introduce a unified contrastive pre-training method to link IAA-style visual and textual patterns. Given the scarcity of text data in real-world scenarios, we also propose the use of composite prompts to guide IAA during the perception training stage, alongside image data.

3.2. The Construction of Aesthetic Descriptions

In previous work, researchers successfully utilized images and corresponding user comments for Image Aesthetics Assessment tasks. However, to our knowledge, the use of MLLMs, such as ChatGPT4, to generate aesthetic descriptions for assisting IAA remains unexplored. As illustrated in Figure 3, we find that human comments are often constrained by subjective perceptions and personal experiences, leading to incomplete or inaccurate descriptions of the visual features and rules of images. While terms like ‘love’ and ‘gorgeous’ reflect individuals’ fondness for images, they are heavily influenced by personal aesthetic preferences and emotions. A photograph may be appreciated by some and disliked by others, making such comments potentially unreliable for aesthetic evaluation. In contrast, MLLMs can generate objective and consistent descriptions by considering various aesthetic attributes and rules. This approach allows for high-quality descriptions even without specialized knowledge in art or photography.

The PARA dataset is a relatively new IAA dataset comprising over 31,000 images. From this dataset, we selected 10,680 images covering 9 topics: ‘Plants’, ‘Night Scenes’, ‘Still Life’, ‘Scenes’, ‘Indoor’, ‘Animals’, ‘Food’, ‘Portraits’, and ‘Architecture’. Using ChatGPT4 with the prompt ‘Analyze the composition and style of the image’, we constructed generative aesthetic descriptions (GADs). As shown in Figure 4, four generated textual descriptions (a, b, c, and d) demonstrate a deep understanding of the aesthetic details and composition of the images. Thus, even without expertise in art or photography, high-quality aesthetic descriptions can be generated using prompts, facilitating easier interpretation of generated textual descriptions to assist in image aesthetic evaluation.

3.3. Unified Contrastive Pre-Training

After constructing the multimodal aesthetics dataset, we use it to complete the unified contrast pre-training stage proposed by our method. CLIP is a model for joint language–image representation. Since CLIP is trained on large-scale general image–text pairs, directly applying it to the IAA domain yields a subpar performance. Inspired by [39] and based on the aesthetic textual descriptions we constructed earlier, we propose a unified contrastive pre-training method for text–image pairs to establish a strong correlation between CLIP and aesthetic tasks. As illustrated in Figure 4, this method allows the CLIP model to better understand aesthetic images and their corresponding generative aesthetic descriptions (GADs).

Specifically, we combine images and GADs into a shared image–text–label space, enabling the CLIP model to utilize both the visual information from the images and the aesthetic data from GADs along with the associated aesthetic labels. This integration allows CLIP to learn richer aesthetic feature representations for transfer learning in the aesthetic domain given the defined aesthetic dataset ${\mathcal{D}}_M$, which comprises N triplet-wise data formats ${\left\{x_i,g_i,y_i\right\}}_{i=0}^N$, where $x_i$ denotes the image, $g_i$ denotes GAD, and $y_i$ denotes the aesthetic label.

Firstly, we utilize the image encoder to process the image $x_i$, and the corresponding GAD $t_i$ is processed by the text encoder, yielding their respective image features $I_i$ and GAD features $T_i$:

$$\left\{\begin{array}{c}{I}_i=f_{\theta }\left(x_i\right),I_i\in R^{d\times 1}\hfill \\ T_i=f_{\phi }\left(g_i\right),T_i\in R^{d\times 1}\hfill \end{array}\right.$$

(1)

For the image in batch $\mathcal{B}$ and the GADs, we normalize their feature vectors using $\mathbb{L}2$ normalization and then map them onto a unit hypersphere. Afterward, we compute the cosine similarity score matrix $Si{m}_{i,GAD}$ for all image–text sample pairs, where the temperature coefficient $\tau$ controls the strength of penalizing hard samples. We consider the bidirectional learning objectives between images and GADs, which can be expressed as follows:

$$\left\{\begin{array}{c}\overline{I_i}={\mathbb{L}}_{2_{norm}}\left(I_i\right)\hfill \\ \overline{T_k}={\mathbb{L}}_{2_{norm}}\left(T_k\right)\hfill \end{array}\right.$$

(2)

$$Si{m}_{i,GAD}=I_i·T_i·exp\left(\tau \right)$$

(3)

We compute the contrastive loss from images to GADs, denoted as ${\mathcal{L}}_{i2GAD}$. Specifically, we apply softmax to the similarity matrix $Si{m}_{i,GAD}$ of each row to obtain the relative probability distribution of each image with respect to all texts in a $\mathcal{B}$. Then, we calculate the corresponding negative log-likelihood loss based on the labels and finally sum and average based on the labels. This can be represented as

$${\mathcal{L}}_{i2GAD}=-\sum _{i\in \mathcal{B}}{\displaystyle \frac{1}{{\left|P\left(i\right)\right|}_{n\in P\left(i\right)}}}log{\displaystyle \frac{exp\left({Sim}_{i,k}\right)}{{\displaystyle \sum _{j\in \mathcal{B}}}exp\left({Sim}_{i,j}\right)}}$$

(4)

where $\mathcal{B}$ denotes a batch of image and GAD data, $Si{m}_{i,j}$ denotes the similarity matrix between images and GADs, and $Si{m}_{i,k}$ denotes image feature $I_i$, the similarity matrix for all GAD features $T_k$ with the same label $y_i$. $P\left(i\right)$ is a set containing the indexes of all GAD features that have the same label $y_i$ as the image $x_i$.

Similarly, we compute the contrastive loss from text to images, denoted as ${\mathcal{L}}_{GAD2i}$. This can be represented as

$${\mathcal{L}}_{GAD2i}=-\sum _{j\in \mathcal{B}}{\displaystyle \frac{1}{{\left|P\left(j\right)\right|}_{n\in P\left(j\right)}}}log{\displaystyle \frac{exp\left({Sim}_{i,k}\right)}{{\displaystyle \sum _{i\in \mathcal{B}}}exp\left({Sim}_{i,j}\right)}}$$

(5)

where $Si{m}_{j,k}$ represents the j-th GAD feature $T_j$, the similarity matrix for all image features $I_k$ with the same label $y_i$. $P\left(j\right)$ is a set containing the indexes of all image features that have the same label $y_i$ as the GAD $T_j$.

Through the unified contrast pre-training method proposed above, the gap between IAA-style visual and textual modes is bridged, thereby improving the model’s ability to understand the complex interactions between aesthetic images and their corresponding GAD.

3.4. Integration of Textual Prompts and Perceptual Training

In real-world scenarios, users’ comments may not be available, and generating aesthetic descriptions might not be convenient. However, after the unified contrastive pre-training process in Section 3.3, the model’s understanding of the complex interactions between aesthetic images and GADs is enhanced, facilitating the transfer to the aesthetic domain. Inspired by [40], we select three prompts—composition, theme, and aesthetic quality—to form a text template to assist in Image Aesthetics Assessment and complete subsequent perception training.

Specifically, we select eight composition techniques: ‘Rule of Thirds’, ‘Negative Space’, ‘Color and Contrast’, ‘Leading Lines’, ‘Depth and Perspective’, ‘Symmetry and Balance’, ‘Focal Point’, and ‘Pattern and Repetition’; the nine themes mentioned in Section 3.2; and five quality ratings: ‘bad’, ‘poor’, ‘fair’, ‘good’, and ‘perfect’ to create combined templates. For instance, “This is a composed photo showing the theme and its aesthetic rating is”, resulting in a total of 360 combinations of prompts $p_i$.

During the perceptual training stage, we concentrate on the task of image aesthetic quality assessment. To enhance the generalization capability and performance of our model, we decide to use different loss functions for optimization. Specifically, our perceptual training framework, as shown in Figure 4, encodes aesthetic images and constructed prompt templates into image embeddings and text embeddings, respectively, based on the model obtained from unified contrastive pre-training. This can be calculated according to Formula (1). Then, the obtained image embeddings and text embeddings are $\mathbb{L}2$ normalized, and the cosine similarity between them is calculated. On one hand, the resulting cosine similarity is processed using Softmax and marginalization to obtain quality logits, which are then weighted and summed to obtain the predicted quality score $S_i$:

$$\widehat{S}\left(x\right)=\sum _{l=1}^{L=5}\widehat{P}(l\mid x)\times l$$

(6)

where L = 5 is the aesthetic quality level; it is necessary to estimate the marginal probability of l. We choose to use a monotonicity-related loss function for optimization. First, we calculate binary labels based on the ground truth (GT) scores of the given aesthetic database. Given an image pair from the IAA database, we convert their corresponding GT values into binary variables of 0 and 1. Then, we optimize using the fidelity loss [41] function.

On the other hand, we effectively fuse the obtained aesthetic quality logits with image embeddings. Since the CLIP architecture is inconsistent between different modalities, we need to fuse vectors of different types of modals to obtain a joint representation. The traditional vector fusion methods are generally dot multiplication, point addition, concat, etc., but these are not as good as vectors because the outer product is more expressive, but the outer product of vectors causes a dramatic increase in dimensionality [42]. Therefore, we introduced a multimodal compact bilinear pooling (MCBP) method to integrate the two different modes $I_i$ and ${Logits}_q$. Specifically, we will use Count Sketch to reduce the dimensionality. Count Sketch is a hashing technique. Through it, we map high-dimensional aesthetic visual features $I_i$ and quality logits ${Logits}_q$ to a lower-dimensional space. This reduces the data dimensionality while preserving the relationships between the original data as much as possible and obtains the features after dimensionality reduction, which can be calculated using the following operations:

$${\widehat{I}}_i,{\widehat{T}}_i=CountSketch\left(I_i,{\mathrm{Logits}}_q\right)$$

(7)

The obtained dimensionality-reduced features are then subjected to a Fast Fourier Transform (FFT) to convert them to the frequency domain. A vector inner product is performed in the frequency domain, followed by an Inverse Fast Fourier Transform (IFFT) to convert them back:

$$M_i=IFFT\left(FFT\left({\widehat{I}}_i\right)\odot FFT\left({\widehat{T}}_i\right)\right)$$

(8)

Finally, perform Signed Sqrt on the obtained time domain features:

$$M_i=SignSqrt\left(M_i\right)$$

(9)

The obtained $M_i$ features undergo fine-grained feature interactions in the multimodal space. Specifically, we optimize the network using different loss functions. For the obtained multimodal features $M_i$, depending on the dataset, we use the EMD loss function for optimization on the AVA dataset and the MSE loss function on the TAD66K and AADB datasets. For the predicted aesthetic quality score Si obtained by weighted summation, we employ the fidelity loss function for optimization:

$$\begin{array}{c}\hfill \mathcal{L}={\mathcal{L}}_{mse/emd}+{\mathcal{L}}_f\end{array}$$

(10)

In our experiments, combining the above losses enables the model to perform well on different datasets and under different evaluation criteria. MSE/EMD losses ensure the accuracy of predictions, while fidelity losses ensure the consistency and rationality of predictions.

4. Experimental Section

4.1. Experimental Datasets

We conducted experiments on three mainstream image aesthetics databases: AVA, AADB, and TAD66K. In the UCP stage, we implemented it based on the PARA database.

AVA [43]: The AVA database is a large-scale dataset containing 250,000 images, each accompanied by approximately 210 user-provided aesthetic scores that range from 1 to 10. We use the training set and test set divided by [43].

AADB [44]: the AADB database consists of 10,000 images annotated by Amazon Mechanical Turk (AMT) workers, including aesthetic score annotation and aesthetic attribute annotation.

TAD66K [2]: the TAD66K database includes 67,125 images across 47 popular themes, with each image receiving detailed annotations from over 1200 people based on specific theme evaluation criteria.

PARA [38]: PARA consists of more than 31,000 images, which was originally designed for personality aesthetics. It has rich annotations, including aesthetic scores, composition scores, emotions, scene categories, etc. We constructed our GADs based on this dataset to perform our UCP.

4.2. Evaluation Metrics

In experiments, we will calculate the performance of the proposed method from seven widely used criteria.

Accuracy (ACC) is an important indicator for evaluating aesthetic binary classification:

$$\begin{array}{c}\hfill \mathrm{ACC}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{P}+\mathrm{N}}}\end{array}$$

(11)

P and N represent high and low aesthetic images, with TP and TN referring to correctly classified high and low aesthetic images, respectively.

For aesthetic regression tasks, common evaluation metrics include the Pearson Linear Correlation Coefficient (PLCC), Spearman Ranked Correlation Coefficient (SRCC), Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Mean Square Error (MSE). The PLCC measures the linear correlation between predicted and actual scores, with higher values indicating better correlation. Formally, the PLCC is defined as

$$\begin{array}{c}\hfill \mathrm{PLCC}={\displaystyle \frac{{\sum }_{i=1}^N\left(x_i-\overline{x}\right)·\left(z_i-\overline{z}\right)}{\sqrt{{\left.{\sum }_{i=1}^N{\left(x_i-\overline{x}\right)}^2·\sqrt{{\sum }_{i=1}^N\left(z_i-\overline{z}\right.}\right)}^2}}}\end{array}$$

(12)

where $\overline{x}$ and $\overline{z}$ represent the average of $x_i$ and $z_i$ on the test set. It is worth noting that before calculating the PLCC, a fractional remapping function needs to be used to compensate for nonlinearity:

$$\begin{array}{c}\hfill f\left(x\right)={\lambda }_1\left({\displaystyle \frac{1}{2}}-{\displaystyle \frac{1}{1+e^{{\lambda }_2\left(x-{\lambda }_3\right)}}}\right)+{\lambda }_4·x+{\lambda }_5\end{array}$$

(13)

where x and $f\left(x\right)$ represent the prediction score and mapping score, respectively, and $\left\{{\lambda }_i|i=1,2,3,4,5\right\}$ are the five parameters determined by fitting [45].

The SRCC evaluates the degree of monotonicity of predictions:

$$\begin{array}{c}\hfill SRCC=1-{\displaystyle \frac{6{\sum }_{i=1}^N{d}_i^2}{N(N^2-1)}}\end{array}$$

(14)

where N is the number of samples and $d_i$ is the difference between the GT and predicted aesthetic scores. The calculation formulas for the RMSE, MSE, and MAE are as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill RMSE& =\sqrt{\frac{1}{N}{\sum }_{i=1}^N{\left(x_i-z_i\right)}^2}\hfill \end{array}\end{array}$$

(15)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill MSE& =\frac{1}{N}{\sum }_{i=1}^N{\left(x_i-z_i\right)}^2\hfill \end{array}\end{array}$$

(16)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill MAE& =\frac{1}{N}{\sum }_{i=1}^N\left|x_i-z_i\right|\hfill \end{array}\end{array}$$

(17)

High SRCC and PLCC values and low MSE, RMSE, and MAE values indicate that this is an excellent IAA model.

The Earth mover’s distance (EMD) is a measure of how close the predicted aesthetic distribution is to the actual aesthetic distribution. For effective IAA tasks, it is best to aim for lower EMD values. It is calculated as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{EMD}={\left(\frac{1}{N_d}{\sum }_{i=1}^{N_d}{\left|CD{F}_y\left(i\right)-CD{F}_{\widehat{y}}\left(i\right)\right|}^r\right)}^{{\displaystyle \frac{1}{r}}}\end{array}$$

(18)

where $CD{F}_y\left(i\right)={\sum }_{i=1}^{N_d}{y}^{\left(i\right)}$ represents the cumulative distribution function. In this article, we set r = 2.

4.3. Implementation Details

Environment: we implemented our method based on PyTorch (v2.5.1) and conducted experiments on an NVIDIA GTX 3090 GPU.

Data preprocessing: During the training stage, images were initially resized to 256 × 256, followed by data augmentation through flipping and random cropping to 224 × 224 for enhanced data variation. During the testing stage, original images were directly resized to 224 × 224 × 3. All input images underwent normalization.

Pre-training strategy: We employed CLIP’s VIT-B/16 as the image encoder and Transformer as the text encoder. Initially, we constructed GADs based on PARA data for unified contrastive pre-training, utilizing the Adam optimizer, initial learning rate of 1 × 10⁻⁵, batch size of 32, and trained for 20 epochs.

Training strategy: In the perception training phase, we formed prompt templates by combining prompts to assist images in IAA tasks. The initial learning rate of 5 × 10⁻⁶ and a cosine annealing scheduler were applied for learning rate adjustment. The batch size was the same as before, and training lasted for 30 epochs. We optimized using L2 loss and fidelity loss on the TAD66K and AADB datasets, while on the AVA dataset, we employed EMD loss and fidelity loss.

4.4. Performance Evaluation with Competing Model

Performance on the AVA: According to the official split [43], we report the performance of the proposed method on the AVA database in

Table 1. Performance comparison of different IAA models in AVA database. “/” means results that were not reported in the original paper. The best results are highlighted in bold.

Method	Backbone	ImageSize	ACC (%)↑	PLCC↑	SRCC↑	MSE↓	RMSE↓	MAE↓	EMD↓
DMA-Net [27]	AlexNet	(227)	75.40	/	/	/	/	/	/
Kong et al. [44]	AlexNet	(227)	77.30	/	0.558	/	/	/	/
APM [46]	ResNet-101	(500)	80.30	/	0.709	0.279	/	/	/
NIMA [17]	VGG16	(224)	80.60	0.610	0.592	/	/	/	0.05
A-Lamp [28]	ResNet-101	(224)	82.50	/	/	/	/	/	/
Zeng et al. [1]	ResNet-101	(384)	80.80	0.720	0.719	0.275	/	/	/
GPF-CNN [47]	InceptionNet	(224)	81.81	0.704	0.690	/	0.524	0.407	0.045
Hosu et al. [48]	Inception-ResNet	Full	81.70	0.757	0.756	/	/	/	/
MUSIQ [49]	ViT	Full	81.50	0.738	0.726	0.242	0.490	/	/
Celona et al. [20]	EfficientNet-B4	Full	81.80	0.732	0.733	/	0.512	0.401	0.044
Niu et al. [50]	ResNet-50	(224)	81.90	0.740	0.734	0.242	/	/	/
TANet [2]	MobileNetV2	(224)	80.60	0.765	0.758	/	/	/	0.047
EAT [51]	Swin Transformer	(224)	/	0.770	0.759	/	/	/	/
EAMB-Net [31]	EfficientNet-B4	(512)	81.36	0.740	0.743	0.258	0.508	0.394	0.043
AesCLIP [23]	ViT	(224)	83.10	0.779	0.771	0.218	/	/	0.041
CADAS [29]	EfficientNet-B4	(224)	82.40	0.751	0.753	/	/	/	0.043
VILA-R [52]	CLIP-B/16	(224)	/	0.774	0.774	/	/	/	/
Gamma [53]	CLIP-B/16	(224)	/	0.749	0.750	/	/	/	/
Ours	CLIP-B/16	(224)	82.40	0.788	0.791	0.218	0.467	0.361	0.040

. AVA is the largest IAA database to date. From Table 1, it can be seen that on the official segmentation, the proposed model using CLIP as the backbone achieves the best performance in most evaluation metrics. For the aesthetic regression task, our model achieves the best results in the SRCC and PLCC. In aesthetic classification tasks and distributions, we achieve a suboptimal and best performance, respectively. These results demonstrate the versatility and effectiveness of our model across various IAA tasks in image aesthetic evaluation.

Performance on the AADB: Most existing methods only report SRCC values, so we evaluate our model on the aesthetic score regression task. The experimental results are summarized in

Table 2. Performance comparison in AADB dataset.

Method	RegNet	DCNN	NIMA	RGNet	Zeng et al.	HIAA	Celona et al.	CADAS	TAVAR	AesCLIP	Ours
	[44]	[54]	[17]	[55]	[1]	[56]	[20]	[29]	[30]	[23]
SRCC↑	0.678	0.689	0.737	0.710	0.719	0.739	0.757	0.768	0.761	0.790	0.795

. Our model achieves the best results, demonstrating the effectiveness of the proposed two-stage training and prompt template construction for IAA.

Performance on the TAD66K: We further conduct experiments on the TAD66K database, which is the recently released database for topic-based IAA. Most existing IAA models have not been experimented on them. We adopt the official partition method, and

Table 3. Performance comparison in TAD66K dataset.

Metric	RAPID	AADB	PAM	NIMA	A-Lamp	HGCN	MLSP	MUSIQ	TANet	Ours
	[26]	[44]	[4]	[17]	[28]	[57]	[48]	[49]	[2]
SRCC↑	0.314	0.379	0.422	0.390	0.411	0.486	0.490	0.517	0.513	0.508
PLCC↑	0.332	0.400	0.440	0.405	0.422	0.493	0.508	0.489	0.531	0.537
MSE↓	0.022	0.021	0.020	0.021	0.019	0.020	0.019	0.018	0.016	0.015

shows our experimental results. We provide competitive results on the TAD66K database. Compared with the listed models, our model achieves the best results in the MSE and PLCC, and the SRCC reaches a suboptimal performance.

4.5. Cross-Database Evaluation

We trained the models using the TAD66K dataset and then evaluated their performance on the AADB dataset without fine-tuning. To further test the model’s generalization ability, we also conducted cross-dataset validation by training on AADB and testing on TAD66K. The results presented in

Table 4. Performance results of cross-dataset validation.

Method	Train on TAD66K and Test on AADB			Train on AADB and Test on TAD66k			Train on AVA and Test on AADB			Train on AADB and Test on AVA
	ACC	PLCC	SRCC	ACC	PLCC	SRCC	ACC	PLCC	SRCC	ACC	PLCC	SRCC
ReLIC	63.2	0.453	0.414	61.3	0.220	0.215	60.9	0.514	0.547	60.6	0.513	0.522
TANet	57.5	0.396	0.392	61.1	0.219	0.211	60.3	0.520	0.599	59.8	0.567	0.564
EMBT	58.9	0.510	0.500	57.0	0.235	0.226	59.9	0.521	0.574	60.7	0.567	0.568
SAAN	59.3	0.313	0.289	60.5	0.193	0.181	61.3	0.556	0.599	61.1	0.583	0.594
Ours	66.8	0.512	0.519	62.9	0.286	0.286	61.4	0.622	0.566	61.3	0.593	0.602

demonstrate that the proposed model maintains a strong generalization performance, effectively adapting to different datasets and performing well across various aesthetic evaluation tasks.

4.6. Ablation Study

Effectiveness of unified contrastive pre-training and perceptual training: We validated the effectiveness of the proposed two training stages on the AVA database. As shown in

Table 5. Ablation experiments of UCP stage and perceptual training stage with prompts AVA.

Unified Contrastive Pre-Training	Prompt for Perceptual Training	PLCC	SRCC	ACC
		0.761	0.757	81.51
✓		0.780	0.778	82.13
	✓	0.784	0.783	81.67
✓	✓	0.788	0.791	82.40

, our CLIP model produced encouraging results after the two-stage pre-training. On the AVA database, we observed a significant performance improvement between CLIP with unified contrastive pre-training and CLIP. The PLCC and SRCC increased by 0.019 and 0.021, respectively, indicating that CLIP successfully transitioned from the general domain to the aesthetic domain after unified contrastive pre-training. During perceptual training, we constructed prompt templates, further enhancing performance on the AVA database. Please note that in the experiments with the CLIP baseline model and CLIP after unified contrastive pre-training, we only used the visual encoder ViT because there was no text input; in the subsequent perceptual training, since we constructed prompt templates to compensate for the unavailability of comments in the real world, we used the image encoder ViT and the text encoder Transformer in the experiments. These results demonstrate that integrating these two training stages to infer image aesthetic scores is necessary and reasonable.

Effectiveness of Multimodal Compact Bilinear Pooling: In this work, we proposed using a multimodal compact bilinear pooling strategy to fuse visual features and quality logits during perceptual training. To investigate the effectiveness of multimodal compact bilinear pooling, we conducted comparative experiments on the AVA, AADB, and TAD66K databases. Specifically, we compared three other fusion strategies—element-wise multiplication, element-wise addition, and concatenation—by replacing the proposed multimodal compact bilinear pooling. For a fair comparison, all comparative experiments were conducted using the same experimental settings.

Table 6. Performance results of the proposed model using different feature fusion blocks.

Method	AVA [43]		AADB [44]		TAD66K [2]
	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC
Concat	0.783	0.783	0.778	0.781	0.529	0.498
Element-wise product	0.781	0.783	0.768	0.783	0.520	0.493
Element-wise addition	0.785	0.784	0.778	0.778	0.523	0.498
MCBP	0.788	0.791	0.782	0.795	0.538	0.508

shows the experimental results in terms of the PLCC/SRCC values. The results indicate that the proposed multimodal compact bilinear pooling outperforms other feature fusion strategies on all databases, further proving that multimodal compact bilinear pooling better captures the interaction information of multimodal aesthetic vectors.

4.7. Visualization

Aesthetic distribution visualization: We visualized the aesthetic distribution predictions on the AVA test set. Figure 5 shows the predicted distributions for some images, demonstrating high accuracy with predicted values closely matching the GT, regardless of aesthetic quality. Additionally, Figure 6 presents scatter plots illustrating the consistency and monotonicity between the true and predicted aesthetic scores, with the r-squared values provided. A high r-squared value, close to the diagonal line, indicates the effectiveness of our model in learning aesthetic distributions.

Attention visualization: We visualized attention maps on AVA. Figure 7 shows the attention maps, where we selected eight thematic categories for attention activation. The maps reveal that our model effectively focuses on key regions across images of different themes, confirming its ability to capture aesthetic attributes.

4.8. Further Discussion

Performance on the art aesthetics domain: The BAID [58] dataset, designed for artistic IAA, contains 60,337 artistic images encompassing various forms of art. To test the applicability of our method in other domains, we performed a performance comparison on the BAID dataset, with results shown in

Table 7. Performance comparison on BAID [58].

Method	PLCC↑	SRCC↑	ACC↑	Params (M)↓
NIMA [17]	0.382	0.393	71.01	23.51
MPada [59]	0.425	0.437	74.33	27.56
MLSP [48]	0.430	0.441	74.92	73.97
BIAA [5]	0.376	0.389	71.61	97.49
TANet [2]	0.437	0.453	75.45	13.88
SAAN [58]	0.467	0.473	76.80	30.81
EAMB [31]	0.487	0.496	77.31	69.03
Ours	0.523	0.505	76.86	82.54

. Our method outperforms others in the SRCC and PLCC, demonstrating its competitiveness and generalizability across domains.

Limitations: Although our proposed method demonstrates an excellent performance, the large parameter size introduced by the CLIP-based model also presents some drawbacks, as shown in Table 7. The substantial number of parameters leads to high consumption of computational resources and storage space during training and inference, thereby extending the training and inference time. Additionally, due to differences in data distribution, applying CLIP to the aesthetic domain requires fine-tuning. In the future, we will focus on reducing model complexity from a lightweight perspective.

5. Conclusions

CLIP has demonstrated its advantages in leveraging cross-modal interactions and has been widely adopted in various downstream tasks. However, due to differences in data distribution, its performance in the aesthetic domain has been less satisfactory. Previous multimodal aesthetic assessment approaches have primarily relied on user reviews, which, while informative, are inherently subjective and vary based on individual preferences. This subjectivity poses challenges in establishing a robust and consistent aesthetic evaluation framework. To address these limitations, we propose a novel multimodal network that transitions from contrastive learning to perceptual training for Image Aesthetics Assessment. Specifically, in the unified contrastive training stage, we leverage a multimodal large language model (MLLM) to generate objective and consistent aesthetic descriptions, mitigating the variability introduced by human subjectivity. Based on this foundation, we utilize image–description–label triples to adapt CLIP from a general domain representation model to an aesthetic-aware model. This novel adaptation process ensures that the learned representations capture the nuanced characteristics of aesthetics beyond standard semantic understanding. Moreover, in real-world applications, user reviews are often unavailable, and generating aesthetic descriptions is impractical in many scenarios. To overcome this challenge, we introduce a combined prompt template for perceptual training, enabling the model to generalize effectively even in the absence of explicit textual feedback. This methodological innovation enhances the model’s adaptability and usability across diverse aesthetic assessment tasks. Extensive experimental results on three popular Image Aesthetic Assessment (IAA) tasks demonstrate the effectiveness of our approach. Our model consistently outperforms existing baselines, showcasing its ability to bridge the gap between human aesthetic perception and automated evaluation. These findings highlight the potential of integrating contrastive learning with perceptual training to advance multimodal aesthetic assessment.