Comparative Evaluation of Multimodal Large Language Models for No-Reference Image Quality Assessment with Authentic Distortions: A Study of OpenAI and Claude.AI Models

Abstract

This study presents a comparative analysis of several multimodal large language models (LLMs) for no-reference image quality assessment, with a particular focus on images containing authentic distortions. We evaluate three models developed by OpenAI and three models from Claude.AI, comparing their performance in estimating image quality without reference images. Our results demonstrate that these LLMs outperform traditional methods based on hand-crafted features. However, more advanced deep learning models, especially those based on deep convolutional networks, surpass LLMs in performance. Notably, we make a unique contribution by publishing the processed outputs of the LLMs, providing a transparent and direct comparison of their quality assessments based solely on the predicted quality scores. This work underscores the potential of multimodal LLMs in image quality evaluation, while also highlighting the continuing advantages of specialized deep learning approaches.

1. Introduction

The proliferation of digital images across various platforms and applications has made automated image quality assessment (IQA) increasingly crucial. While traditional IQA methods often rely on reference images to evaluate quality, real-world scenarios frequently demand no-reference image quality assessment (NR-IQA), where quality must be determined without access to pristine versions of the images [1]. This challenge is particularly pronounced when dealing with authentic distortions that occur naturally during image capture and processing, as opposed to synthetic distortions commonly used in controlled studies [2]. Real-world scenarios often involve diverse and unpredictable distortions, making it challenging to assess image quality accurately. Non-distortion-specific algorithms, which do not require prior knowledge of distortion types, are increasingly important for practical applications [3]. In medical imaging, image quality directly affects diagnostic accuracy. Quality assurance processes, including acceptance testing and periodic checkups, are essential to ensure high-quality radiographs with minimal radiation exposure [4]. In an autonomous driving system, camera sensors acquire images from highly diverse and complex environments [5], and the quality of the camera images directly affects the performance of object detection [6], image classification [7], and image segmentation [8] algorithms. Digital image quality assessment may also have practical applications in surveillance and security [9], video streaming services [10], astronomical image processing [11], manufacturing quality control [12], and remote sensing [13].

Recent advances in vision–language models have revolutionized various computer vision tasks through their ability to understand both visual content and natural language descriptions. Large-scale models such as GPT-4V, Claude Vision, and PaLM-V have demonstrated remarkable capabilities in processing visual information alongside textual context. These models employ sophisticated architectures: GPT-4V utilizes a vision transformer (ViT) backbone with specialized attention mechanisms for cross-modal interaction, while Claude Vision implements a novel architecture combining convolutional and transformer layers with enhanced spatial reasoning capabilities. PaLM-V builds upon the PaLM language model architecture [14], incorporating visual tokens through a perception system that maintains high-resolution feature maps. These models are typically trained on massive datasets exceeding billions of image–text pairs, such as LAION-5B [15], WebLI [16], and Conceptual Captions [17], enabling them to learn rich semantic representations that bridge visual and linguistic information.

Traditional NR-IQA approaches have evolved significantly over the past decade. Hand-crafted feature methods began with Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) [18], which utilizes natural scene statistics (NSS) in the spatial domain, analyzing mean-subtracted contrast-normalized (MSCN) coefficients [19] and their pairwise products. Naturalness Image Quality Evaluator (NIQE) [20] advanced this approach by modeling multivariate Gaussian distributions of space-domain NSS. Deep learning approaches introduced new paradigms: CNNIQA pioneered the use of convolutional neural networks with a novel regression strategy, while NIMA [21] implemented a novel architecture using MobileNet as backbone, trained on the AVA (Aesthetic Visual Analysis) [22] dataset with earth mover’s distance (EMD) [23] loss. MetaIQA further advanced the field by incorporating meta-learning techniques to improve generalization across different distortion types. Recent developments include MUSIQ [24], which handles multiple scales through transformer architectures, and HyperIQA [25], which employs hypernetworks to generate quality-aware features. These methods have primarily been evaluated on synthetic distortion datasets such as LIVE [26], TID2013 [27], and KADID-10k [28].

The challenge of authentic distortions represents a significant hurdle in real-world IQA applications. Unlike synthetic distortions, authentic distortions present complex, often interconnected degradations [29,30,31,32]:

• Motion blur from camera shake or subject movement, characterized by directional smearing and varying intensity along the motion path;
• Poor focus, manifesting as both global and local blur patterns, often with depth-dependent characteristics;
• Exposure issues including highlight clipping, shadow noise, and dynamic range limitations;
• Sensor noise varying with ISO settings and showing color-channel-dependent characteristics;
• Optical aberrations including chromatic aberration, barrel/pincushion distortion, and vignetting;
• Complex interactions between multiple distortion types, such as noise amplification in post-processed underexposed regions.

Unlike conventional NR-IQA models that focus on low-level features, multimodal LLMs can contextualize image quality based on semantic content. For example, they may interpret whether blurring affects a face differently than a background [32] or how compression artifacts alter readability in text-heavy images [33]. Pretrained multimodal LLMs have seen vast amounts of real-world imagery, allowing them to generalize across diverse distortions (e.g., noise, blur, and compression) and datasets, rather than being restricted to specific distortions present in training data. Since these models can process both visual and textual inputs, they can be adapted to novel NR-IQA tasks, such as aesthetic quality assessment, perceptual similarity, or domain-specific evaluations (e.g., medical images and satellite imagery). Despite their potential, multimodal LLMs are not explicitly optimized for NR-IQA and may lack fine-grained sensitivity to certain distortions.

1.1. Contributions

This work systematically evaluates the capability of multimodal LLMs in the task of NR-IQA for authentic distortions. Our main contributions are as follows:

• We assess the ability of state-of-the-art LLMs to predict image quality scores on two widely used IQA benchmark datasets: CLIVE [34] and KonIQ-10k [1].
• We investigate the frequency of invalid responses across different models and datasets, identifying systematic weaknesses.
• We compare the performance of multiple multimodal LLMs, including ChatGPT-4o-Latest, GPT-4o-2024-11-20, GPT-4-Turbo-2024-04-09, Claude-3-Haiku-20240307, Claude-3-Opus-20240229, and Claude-3-Sonnet-20240229, highlighting variations in their robustness and accuracy. Our results reveal that LLMs struggle more with the CLIVE [34] dataset than with KonIQ-10k [1], suggesting dataset-specific challenges that affect multimodal LLM performance.

1.2. Structure of the Paper

The remainder of this paper is organized as follows. Section 2 provides an overview of previous studies on NR-IQA, including both traditional and deep learning-based approaches. Section 3 describes the datasets used for evaluation, the experimental setup, and the methodology for assessing the performance of different LLMs in NR-IQA. Section 4 presents a performance comparison of various LLMs against traditional and deep learning-based NR-IQA methods. The section includes quantitative evaluations in the form of statistical correlations with ground-truth scores. Section 5 interprets the findings, highlights key observations, and compares LLM performance across different architectures. It also discusses potential limitations and implications for real-world applications. Section 6 summarizes the key contributions of this study and suggests directions for future research in integrating LLMs into NR-IQA tasks.

2. Related Works

NR-IQA has emerged as a critical area of research within the field of computer vision, particularly due to its applicability in scenarios where reference images are unavailable, such as streaming applications [10] and autonomous vehicles [35]. NR-IQA aims to evaluate the perceptual quality of a given image based solely on the distorted image itself, making it a challenging task due to the absence of a pristine (or reference) image for comparison [36]. The complexity of NR-IQA arises from the diverse types of distortions and the inherent variability of image content, necessitating the development of robust algorithms capable of generalizing across different scenarios [37,38].

No-reference image quality assessment methods that do not rely on deep learning techniques often utilize various image features, statistical models, and computational algorithms to evaluate image quality without the need for a reference image. A popular approach is natural scene statistics (NSS), which refers to the statistical properties found in natural images, capturing the regularities and patterns commonly observed in the visual environment [39]. NSS analysis aims to understand and model the statistical distribution of pixel intensities, gradients, textures, and other visual features in natural scenes [40]. These statistical properties provide valuable insights into the characteristics of typical images encountered in the real world. NSS analysis has been widely used in the field of image processing and computer vision, particularly in tasks related to image quality assessment and image compression. By understanding the statistical regularities present in natural scenes, researchers can design algorithms that enhance the quality of processed images or improve the efficiency of image compression techniques [41]. In the context of image quality assessment, NSS-based methods analyze the statistical features of an image or a region within an image. These features may include moments, histograms, or other statistical measures derived from pixel intensities, gradients, or texture patterns. For instance, Moorthy and Bovik [42] introduced an algorithm that predicts the presence of a set of predefined distortions, i.e., JPEG compression noise, JPEG2000 compression noise, white noise, Gaussian blur, and fast fading, in an image. Next, perceptual quality scores are obtained along each of the predefined distortions. Finally, the quality of the input image is estimated as a probability-weighted summation. Quality-aware NSS features were extracted from Daubechies 9/7 wavelet subband coefficients [43] by fitting a generalized Gaussian distribution (GGD) onto them. Further, the parameters of the GGD were used as quality-aware features, which were mapped onto quality ratings with a trained support vector regressor (SVR) [44]. In [45], the authors also utilized wavelet coefficients, but their distributions were modeled using a Gaussian scale mixture [46]. In contrast, Saad et al. [47] defined NSS in the discrete cosine transform (DCT) domain. As in [42], the distribution of DCT coefficients was modeled with a GGD, and its shape parameters were taken as quality-aware features. Finally, these features were utilized in a Bayesian inference method for estimating perceptual image quality. A similar approach was presented in [48], where four quality-aware features were considered in the DCT domain, i.e., kurtosis, contrast, and two anisotropy measures [49]. Contrarily, Mittal et al. [18] fitted an asymmetric GGD [50] onto the normalized luminance coefficients [51] of a distorted image, and quality-aware features were obtained from the asymmetric GGD’s parameters. Besides NSS-based features, texture features [52] were also utilized for NR-IQA [3], since image texture is heavily influenced by the presence of noise. For instance, Xue et al. [53] utilized the gradient magnitude maps and the Laplacian of Gaussian responses of images to extract structural information. Local binary patterns (LBPs) [54], which are probably the most popular texture descriptors in the computer vision literature [55], have also been intensively applied for NR-IQA. In [56], Freitas et al. gave an overview of a wide range of LBP variants and their performance in NR-IQA. In [57,58], the authors combined LBP-based structural features with other types of features, i.e., statistical and luminance features, to achieve improved performance.

Recent advancements in deep learning have significantly influenced the development of NR-IQA methodologies. For instance, various neural network architectures, including convolutional neural networks (CNNs) and generative adversarial networks (GANs), have been employed to enhance the feature extraction capabilities of NR-IQA systems [59,60]. These approaches leverage the ability of deep learning models to learn hierarchical representations of image features, which can be crucial for accurately predicting image quality scores. The work of Kang et al. [61] was a pioneering achievement in the application of CNNs for NR-IQA. Specifically, the authors trained a CNN from scratch on image patches with dimensions of $32\times 32$ (the patches inherited the source images’ perceptual quality). The perceptual quality of a test image was determined by taking the arithmetic average of the sampled patches’ quality ratings. Similarly, Bare et al. [62] proposed an end-to-end CNN-based solution trained on image patches, but the patches’ target values did not inherit the quality score of the source image, since the target values were determined using an image similarity metric [63]. In contrast, Li et al. [64] modified and fine-tuned on the ImageNet [65] database trained Network in Network (NiN) [66] architecture. The NiN was modified by inserting five new layers after the 26th to make it suitable for NR-IQA. Next, it was completely retrained. Another line of papers [67,68,69] focused on feature extraction with ImageNet-pretrained CNNs, and the extracted features were mapped onto perceptual quality using a regression technique, typically an SVR. The main advantage of this scheme is that an image quality estimator can be implemented without training a neural network but while still relying on the semantic information stored in the CNN. On the other hand, they contain two steps, namely, deep feature extraction and regression, thus requiring the storage of long feature vectors. As a consequence, they are rather suitable for small- or medium-sized databases. For instance, Gao et al. [70] extracted deep features from all 37 layers of a very large pretrained CNN. Next, the deep features of each layer were mapped onto quality scores with SVRs. Finally, the arithmetic mean of the quality ratings provided by each layer was considered to represent a digital image’s perceptual quality. Besides CNNs, transformer neural networks have also become a hot research topic in the deep learning community [71]. Unlike CNNs, transformers rely entirely on self-attention mechanisms to capture relationships between input tokens, making them particularly effective for tasks involving sequential or positional data, such as language translation and text generation [72]. The key innovation in transformers is the self-attention mechanism, which allows the model to weigh the significance of different input tokens when making predictions. This mechanism enables architectures leading to better performance in various natural language processing (NLP) tasks. Although transformers were initially introduced for NLP, researchers have explored their applicability in the field of computer vision. One notable architecture that combines transformers with computer vision is Vision Transformer (ViT) [71]. In ViT, an image is divided into fixed-size non-overlapping patches, which are treated as tokens similar to words in NLP. These patches are then flattened and linearly embedded to form input embeddings for the transformer model. The transformer processes these embeddings using self-attention mechanisms, allowing the model to capture relationships between different image patches. For NR-IQA, Ke et al. [24] implemented a transformer-based method that uses hash-based 2D absolute position encoding for the embedding of image patches at multiple scales. Similarly, the authors of [73] applied multi-scale feature extraction to achieve improved performance. In contrast, Yang et al. [74] first applied ViT for feature extraction and then introduced two specific blocks to strenghten local and global interactions. Wang et al. [75] utilized CLIP (Contrastive Language–Image Pre-training) [76], which is a widely used multimodal LLM, to evaluate the similarity between an image and a prompt such as $\{“Goodphoto.”,“Badphoto.”\}$. Furthermore, the researchers identified several descriptive features, i.e., brightness and colorfulness, which can be estimated using this prompt strategy.

Since NR-IQA is a popular research topic and researchers have proposed many different methods over the years, an encyclopedic summary of NR-IQA methods would be outside the scope the scope of this study. In [77], Manap et al. presented a survey in which NSS-based NR-IQA methods were thoroughly discussed. Similarly, Xu et al. [78] introduced this topic from the point of view of quality-aware features. In [79], the authors discussed methods relying on deep learning techniques and several emerging topics in IQA, such as quality assessment of screen content images and 3D IQA. In [80], Yang et al. compared several deep learning-based algorithms to each other. An analysis of the current state of the art of NR-IQA can be found in the book of Xu et al. [81], and the PhD dissertations of Jenadeleh [82] and Men [83] can also be considered important works in the NR-IQA literature.

3. Materials and Methods

3.1. Materials

3.1.1. IQA Benchmark Databases

The evaluation of IQA models requires robust and diverse benchmark datasets that capture a wide range of distortions and subjective human opinions. Two commonly used databases for IQA research are CLIVE [34] and KonIQ-10k [1]. Both databases are widely recognized for their content diversity, real-world distortions, and high-quality subjective annotations, making them essential benchmarks for deep learning-based IQA models.

The CLIVE [34] database was developed to evaluate the performance of IQA models on images containing authentic, real-world distortions. Unlike artificially distorted datasets, CLIVE focuses on complex and diverse distortions encountered in practical scenarios. The CLIVE database consists of 1169 images that were collected using mobile devices. These images are characterized by diverse distortions, including motion blur, overexposure, underexposure, color artifacts, and noise. The wide range of real-world imperfections makes CLIVE a challenging benchmark for IQA models. To assess perceived image quality, subjective evaluations were gathered through crowdsourcing experiments, where a large number of participants rated the images based on their visual quality. The ratings were then aggregated to form mean opinion scores (MOSs), which serve as ground-truth labels for training and evaluating IQA models. The authentic distortions in CLIVE reflect real-world imaging challenges, such as those encountered in mobile photography. This makes CLIVE an essential dataset for developing and testing models designed to handle natural, unpredictable image degradation.

The KonIQ-10k [1] database was created to support research on image quality assessment, with a focus on providing a large and diverse set of high-quality images paired with subjective quality annotations. KonIQ-10k contains 10,073 images, which were selected from the Yahoo-Flickr Creative Commons 100 Million [84] (YFCC100M) dataset. The images cover a wide range of scenes and content types, including landscapes, people, architecture, and abstract patterns. The images were resized to 1024 × 768 pixels, ensuring consistency in resolution while maintaining visual richness. Image quality ratings for KonIQ-10k were collected through a large-scale crowdsourcing campaign, where participants provided their subjective assessments. The collected ratings were processed to generate MOSs, ensuring high reliability and consistency. To reduce bias, the experiment was carefully designed, considering factors such as participant diversity and the rating environment. KonIQ-10k is notable for its high content diversity and high-quality subjective labels. The dataset offers a comprehensive benchmark for evaluating the generalization ability of IQA models across various types of content and distortions. Additionally, the large-scale nature of the dataset (over 10,000 images) provides a robust platform for training deep learning models.

Both CLIVE [34] and KonIQ-10k [1] databases are widely used in the field of image quality assessment. While CLIVE emphasizes real-world, authentic distortions collected through mobile imaging, KonIQ-10k offers a broader range of content diversity with carefully curated subjective scores from a large-scale crowdsourcing effort [85]. Together, these datasets serve as complementary benchmarks for evaluating and improving deep learning-based IQA models, highlighting their ability to generalize across different types of distortions and content.

Table 1. Applied benchmark databases.

	CLIVE [34]	KonIQ-10k [1]
Publication year	2015	2018
Number of images	1169	10,073
Subjective framework	Crowdsourcing	Crowdsourcing
Number of annotators	8000	350,000
Number of annotations	1400	1,200,000
Resolution	$500\times 500$	$1024\times 768$
Quality score range	0–100	1–5

provides an overview of the benchmark databases used for evaluating the image quality assessment performance of multimodal LLMs. Figure 1 illustrates the relative distributions of MOS for the CLIVE [34] and KonIQ-10k [1] benchmark IQA databases. Figure 2 presents a selection of images from the CLIVE benchmark dataset, each annotated with an MOS displayed in a yellow box.

3.1.2. Multimodal LLMs

OpenAI has introduced a range of multimodal LLMs that integrate text, image, and reasoning capabilities. The lineup includes ChatGPT-4o-Latest, GPT-4o-2024-11-20, and GPT-4-Turbo-2024-04-09, each designed to address distinct use cases while maintaining strong performance across multimodal tasks. ChatGPT-4o-Latest is the most recent iteration of OpenAI’s flagship model, optimized for real-time conversational AI and multimodal reasoning. It is able to process text and images simultaneously, making it suitable for complex dialogue systems, visual question answering (VQA), and multimodal search [86]. It is characterized by strong generalization across tasks with improved reasoning over images and text, delivering high accuracy on multimodal benchmarks. GPT-4o-2024-11-20 is a variant of GPT-4o designed with enhanced performance in multimodal content generation and analysis. It excels in document understanding, chart analysis, and image-based content creation, integrating detailed text generation with visual context interpretation. It demonstrates high performance in benchmarks for image–text reasoning and document analysis. GPT-4-Turbo-2024-04-09 is a performance-optimized version of GPT-4, tailored for cost-efficient deployment without compromising quality [87]. It is designed for general-purpose multimodal tasks such as image description, automated reporting, and conversational assistance. It offers a balance between cost efficiency and performance, making it suitable for large-scale enterprise deployments.

The Claude series of multimodal LLMs, developed by Anthropic, represents a significant advancement in AI-driven reasoning and image analysis. The series includes three models: Claude 3 Haiku, Claude 3 Opus, and Claude 3 Sonnet, each tailored for different use cases while sharing a common architecture designed for high-performance multimodal tasks [88]. Haiku is the fastest and most lightweight model in the Claude 3 series, optimized for real-time inference and large-scale deployments. Despite its compact size, it can process both text and image inputs effectively, making it suitable for quick, scalable tasks such as content moderation, chatbot interactions, and real-time image analysis. It is characterized by high-speed processing and low latency, ideal for applications requiring real-time responses. Opus is the flagship model, designed for complex reasoning tasks and high-precision multimodal analysis. It excels in understanding and generating text, as well as analyzing visual inputs, making it highly effective for tasks such as image captioning, document analysis, and complex reasoning over charts or infographics. Sonnet is the mid-tier model, offering a balance between the speed of Haiku and the high performance of Opus. Designed for general-purpose multimodal tasks, it provides strong performance across a wide range of applications, including customer support automation, image-based queries, and document processing. It is a balanced trade-off between performance and efficiency, making it a versatile choice for diverse applications. Collectively, these models highlight the capabilities of LLMs in combining textual and visual understanding to address a broad spectrum of AI challenges [89].

To provide a more balanced and comprehensive evaluation, we included LLaVA-1.5-7B-HF [90], a state-of-the-art open-source multimodal LLM. While proprietary systems, such as Claude, offer impressive performance, they operate as black boxes with limited transparency regarding architecture, training data, and inference mechanisms. In contrast, open-source models enable reproducibility, deeper analysis, and community-driven improvements. The model architecture consists of the following components:

• Vision Encoder: LLaVA-1.5-7B’s image encoder is CLIP ViT-L/14, which transforms an input image into a sequence of visual embeddings. CLIP (Contrastive Language–Image Pre-training) was trained on 400 million image–text pairs to produce aligned visual and textual representations.
• Language Model: The text processing backbone is Vicuna-7B v1.5, a fine-tuned version of LLaMA-2 7B optimized for instruction-following dialogue through supervised fine-tuning on high-quality multi-turn conversation data.
• Projection Layer: A trainable linear layer maps the output of the vision encoder to the language model’s embedding space. This enables the integration of visual information into the language model’s input stream.
• Training Paradigm: The model was trained using instruction-tuning on a mixture of datasets, including complex visual reasoning, image captioning, and visual question answering (VQA). The fine-tuning process includes aligning the model to follow multimodal prompts using image and text inputs jointly.
• Tokenization and Input Format: Images are encoded into visual tokens and prepended to the textual input as special tokens. During inference, a multimodal prompt (e.g., “Describe the quality of this image”) is concatenated with the visual tokens and fed into the model for autoregressive decoding.
• Model Availability: The “HF” suffix denotes the Hugging Face-compatible implementation, which includes pre-trained weights, configuration files, and inference scripts. It can be run on consumer-grade GPUs and integrated into custom pipelines.

3.2. Methods

As already mentioned, this study evaluates the ability of multimodal LLMs to perform NR-IQA on digital images with authentic distortions. Each model was prompted with a standardized question requesting an overall quality rating for each image. The question was formulated as follows for CLIVE [34]: “How would you rate the overall visual quality of this image on a continuous scale of 0.0 to 100.0, where 0.0 represents the lowest quality and 100.0 represents the highest quality?” For KonIQ-10k [1] benchmark database, a different rating scale was used: “How would you rate the overall visual quality of this image on a continuous scale of 1.0 to 5.0, where 1.0 represents the lowest quality and 5.0 represents the highest quality?” As already mentioned, the following multimodal LLMs were tested in this study: ChatGPT-4o-Latest, GPT-4o-2024-11-20, GPT-4-Turbo-2024-04-09, Claude-3-Haiku-20240307, Claude-3-Opus-20240229, and Claude-3-Sonnet-20240229. Model predictions were assessed by computing the correlation between the predicted scores and the ground-truth scores. However, in some instances, the models refused to provide an evaluation, citing privacy concerns or content restrictions. The frequencies of such invalid responses varied across models and are depicted in Figure 3. GPT-4o-2024-11-20 exhibited the highest invalid response rate, particularly for CLIVE images, whereas GPT-4-Turbo-2024-04-09 produced no refusals on KonIQ-10k. ChatGPT-4o-Latest also showed a relatively high invalid response rate for CLIVE (1.03%), but the rate remained low (0.04%) for KonIQ-10k. Claude-3-Haiku-20240307 had moderate invalid response rates for both datasets, at 0.68% for CLIVE and 0.53% for KonIQ-10k. Claude-3-Opus-20240229 had the lowest rates overall, showing the best robustness (0.00% for CLIVE and 0.13% for KonIQ-10k). It stands out as the most reliable model, producing almost no invalid responses. GPT-4-Turbo-2024-04-09 performs well, with a very low invalid response rate (0.09% for CLIVE and 0.00% for KonIQ-10k). The difference in refusal rates between datasets suggests that model responses may be influenced by the content characteristics of each dataset. CLIVE [34] has a higher invalid response rate across most models, suggesting that this dataset may be more challenging for multimodal LLMs than KonIQ-10k [1]. While evaluating LLaVA-1.5-7B-HF [90], we observed a high rate of invalid or uninformative responses in the context of NR-IQA. Unlike proprietary models, which typically provide a structured refusal along with a justification—often citing potential safety or misuse concerns—LLaVA tends to fail silently. In many cases, instead of outputting a quality score or indicating that it cannot perform the task, it defaults to generic image descriptions unrelated to perceptual quality. This lack of explicit refusal or scoring degrades the model’s reliability and makes it difficult to distinguish between true low-confidence cases and fundamental task limitations.

4. Results

4.1. Evaluation Metrics

In research studies, the performance of an NR-IQA method is given by the correlation strength calculated between predicted and ground-truth quality scores. To this end, Pearson’s linear correlation coefficient (PLCC), Spearman’s rank order correlation coefficient (SROCC), and Kendall’s rank order correlation coefficient (KROCC) are usually given. They are also adopted in this paper. PLCC is a statistical measure that quantifies the strength and direction of a linear relationship between two variables. It ranges from $-1$ to 1, where $PLCC=1$ indicates a perfect positive linear correlation: as one variable increases, the other variable increases proportionally. In contrast, $PLCC=-1$ indicates a perfect negative linear correlation. Finally, $PLCC=0$ indicates no linear correlation. PLCC is calculated using the following formula:

$$PLCC(X,Y)={\displaystyle \frac{\sum (X_i-\overline{X})\sum (Y_i-\overline{Y})}{\sqrt{\sum {(X_i-\overline{X})}^2\sum {(Y_i-\overline{Y})}^2}}},$$

(1)

where $X_i$ and $Y_i$ are the individual data points in the two variables, while $\overline{X}$ and $\overline{Y}$ are the means of the respective variables. As recommended in [26], prior to the calculation of PLCC, a non-linear mapping is carried out between the predicted and ground-truth scores using a logistic function with five parameters. SROCC is a non-parametric measure that assesses the strength and direction of a monotonic relationship between two variables. Unlike PLCC, SROCC does not assume a linear relationship between the variables. The calculation of SROCC involves the following steps: (i) Rank the values of each variable separately, assigning ranks from 1 to n, where n is the number of data points. (ii) Calculate the difference between the ranks for each pair of corresponding data points. (iii) Square these differences. (iv) Sum up these squared differences to obtain the sum of squared differences. SROCC can be expressed using the following formula:

$$SROCC(X,Y)=1-{\displaystyle \frac{6\sum d_i^2}{n(n^2-1)}},$$

(2)

where $d_i$ is the difference between the ranks of the corresponding pairs of data points and n is the number of data points. SROCC, like PLCC, ranges from $-1$ to 1. $SROCC=1$ indicates a perfect positive monotonic correlation, $SROCC=-1$ indicates a perfect negative monotonic correlation, and $SROCC=0$ indicates no monotonic correlation. SROCC provides a measure of how well the relationship between two variables can be described using a monotonic function, regardless of the specific shape of that function. KROCC is also a non-parametric measure used to assess the strength and direction of a monotonic relationship between two variables. Like SROCC, KROCC is suitable for ordinal and ranked data and does not assume a linear relationship. The calculation of KROCC involves counting concordant and discordant pairs of observations. Then, KROCC can be expressed using the following formula:

$$KROCC(X,Y)={\displaystyle \frac{C-D}{\left(\genfrac{}{}{0pt}{}{n}{2}\right)}},$$

(3)

where C and D denote the numbers of concordant and discordant pairs, respectively, between X and Y.

4.2. Performance Comparison of Multimodal LLMs

Figure 4 compares the performance of various multimodal LLMs on NR-IQA using the CLIVE [34] dataset. From the results, ChatGPT-4o-Latest and GPT-4o-2024-11-20 consistently outperform the other models, showing the highest correlation with human quality ratings. GPT-4-Turbo-2024-04-09 also demonstrates strong performance, although it slightly underperforms top two models. In contrast, the Claude-series models (Haiku, Opus, and Sonnet) exhibit noticeably weaker performance, particularly Claude-3-Haiku-20240307, which receives the lowest scores across all metrics. This suggests that OpenAI’s models are significantly more aligned with human judgments on this task than Anthropic’s models are. The performance gap between the best and worst models is substantial, indicating that not all LLMs are equally effective for image quality assessment tasks. Figure 5 presents a comparative analysis of multimodal LLMs for NR-IQA on the KonIQ-10k [1] dataset.

Table 2. Performance of LLaVA-1.5-7B-HF [90] on two benchmark NR-IQA datasets. The table reports correlation metrics between the model’s predicted quality scores (when available) and the ground-truth quality scores.

	CLIVE [34]	KonIQ-10k [1]
PLCC	0.098	0.111
SROCC	0.082	0.098
KROCC	0.061	0.069

shows the performance of LLaVA-1.5-7B-HF [90] on the CLIVE [34] and KonIQ-10k [1] benchmarks. The correlation scores with ground-truth quality ratings are notably low, indicating limited effectiveness in NR-IQA tasks. This contrasts with the proprietary models evaluated in the previous section, which achieve substantially higher correlations, highlighting a significant performance gap between open-source and commercial multimodal systems.

Figure 6 and Figure 7 present scatter plots comparing ground-truth image quality scores with predicted scores from the examined multimodal LLMs on the CLIVE and KonIQ-10k benchmark datasets, respectively. Models with better predictive performance should have points more closely aligned with the diagonal. Figure 8 and Figure 9 present histograms of quality ratings assigned by six multimodal LLMs on the CLIVE [34] and KonIQ-10k [1] benchmark datasets, respectively. Figure 10 and Figure 11 display residual plots for six multimodal LLMs on the CLIVE [34] and KonIQ-10k [1] benchmark databases, respectively. Most models exhibit a systematic pattern in their residuals, suggesting potential biases in their predictions. Specifically, Claude-3-Haiku-20240307 shows the most distinct pattern, with a strong negative slope, meaning it consistently overpredicts across most of the range, with this overprediction becoming more severe as ground-truth values increase. Claude-3-Sonnet-20240229 shows a less defined pattern with greater scatter but still tends toward overprediction, especially for higher ground-truth values. The other models show more balanced error distributions around the zero line but have similar patterns, with positive slope in residuals, suggesting they tend to underpredict high values and overpredict low values. All models show horizontal striping, suggesting the models may be quantizing their predictions to certain values or intervals. All models struggle more with extreme values (very low or very high ground truths).

4.3. Comparison to the State-of-the-Art

Table 3. Comparison to traditional state-of-the-art methods on CLIVE [34] and KonIQ-10k [1] databases. Median PLCC, SROCC, KROCC values were obtained over 100 random train–test splits. Best values are in red, second-best values in green, and third-best values in blue.

	CLIVE [34]			KonIQ-10k [1]
Method	PLCC	SROCC	KROCC	PLCC	SROCC	KROCC
BIQI [42]	0.519	0.488	0.329	0.688	0.662	0.471
BLIINDS-II [47]	0.473	0.442	0.291	0.574	0.575	0.414
BMPRI [92]	0.541	0.487	0.333	0.637	0.619	0.421
BRISQUE [18]	0.524	0.497	0.345	0.707	0.677	0.494
CurveletQA [93]	0.636	0.621	0.421	0.730	0.718	0.495
DIIVINE [45]	0.617	0.580	0.405	0.709	0.693	0.471
ENIQA [94]	0.596	0.564	0.376	0.761	0.745	0.544
GM-LOG-BIQA [53]	0.607	0.604	0.383	0.705	0.696	0.501
GWH-GLBP [95]	0.584	0.559	0.395	0.723	0.698	0.507
IL-NIQE [96]	0.487	0.415	0.280	0.463	0.447	0.306
NBIQA [97]	0.629	0.604	0.427	0.771	0.749	0.515
NIQE [20]	0.328	0.299	0.200	0.319	0.400	0.272
PIQE [98]	0.172	0.108	0.081	0.208	0.246	0.172
OG-IQA [99]	0.545	0.505	0.364	0.652	0.635	0.447
Robust BRISQUE [100]	0.522	0.484	0.330	0.718	0.668	0.477
SSEQ [101]	0.487	0.436	0.309	0.589	0.572	0.423
ChatGPT-4o-Latest	0.758	0.732	0.576	0.805	0.760	0.610
GPT-4o-2024-11-20	0.775	0.745	0.588	0.807	0.765	0.618
GPT-4-Turbo-2024-04-09	0.696	0.677	0.520	0.729	0.694	0.535
Claude-3-Haiku-20240307	0.484	0.474	0.353	0.458	0.370	0.291
Claude-3-Opus-20240229	0.557	0.529	0.396	0.683	0.602	0.459
Claude-3-Sonnet-20240229	0.610	0.586	0.437	0.717	0.664	0.517

presents a comparative performance analysis of various NR-IQA methods based on hand-crafted features and various multimodal LLMs. As already mentioned, the performance of each method is quantified using three widely used correlation metrics: PLCC, SROCC, KROCC. These values were computed as the median over 100 random train–test splits in the case of NR-IQA algorithms based on hand-crafted features. To obtain the performance metrics, the original source codes published by the researchers were used in the test process. In contrast, multimodal LLMs were evaluated in a zero-shot setting by prompting each model once for the entire dataset without additional training or adaptation. The results indicate that multimodal LLMs—particularly OpenAI’s GPT-4o models—outperform traditional hand-crafted feature-based NR-IQA methods and achieve competitive performance compared to some deep learning-based approaches. Among the evaluated models, GPT-4o-2024-11-20 achieved the highest performance on both datasets, with PLCC, SROCC, and KROCC values of 0.775, 0.745, and 0.588 on CLIVE and 0.807, 0.765, and 0.618 on KonIQ-10k, respectively. The latest GPT-4o variant also performed well, securing the second-best results in most cases. In contrast, Claude-3 models exhibited significantly lower correlation scores, with Claude-3-Haiku achieving the weakest performance overall, while some Claude-3 models demonstrated moderate correlation with human judgments, they were consistently outperformed by OpenAI’s models across all metrics. Similar to Table 3,

Table 4. Comparison to deep state-of-the-art methods on CLIVE [34] and KonIQ-10k [1] databases. Median PLCC, SROCC, KROCC values were obtained over 100 random train–test splits. Best values are in red, second-best values in green, and third-best values in blue. If the data are not available, it is denoted by ‘-’.

	CLIVE [34]			KonIQ-10k [1]
Method	PLCC	SROCC	KROCC	PLCC	SROCC	KROCC
BLIINDER [70]	0.782	0.763	0.576	0.876	0.864	0.668
UNIQUE [102]	0.891	0.855	0.633	0.900	0.897	0.664
CONTRIQUE [103]	0.857	0.845	-	0.906	0.894	-
DB-CNN [104]	0.869	0.851	-	0.884	0.875	-
DeepFL-IQA [105]	0.769	0.734	-	0.887	0.877	-
HyperIQA [25]	0.882	0.859	-	0.917	0.906	-
KonCept512 [106,107]	0.848	0.825	-	0.937	0.921	-
MAMIQA [91]	0.895	0.874	-	0.937	0.926	-
MLSP [105,108]	0.769	0.734	-	0.887	0.877	-
PaQ-2-PiQ [109]	0.850	0.840	-	0.880	0.870	-
PQR [110]	0.882	0.857	-	0.884	0.880	-
VCRNet [111]	0.865	0.856	-	0.909	0.894	-
WSP [112]	-	-	-	0.931	0.918	-
ZEN-IQA [113]	0.664	0.672	-	0.796	0.776	-
ChatGPT-4o-Latest	0.758	0.732	0.576	0.805	0.760	0.610
GPT-4o-2024-11-20	0.775	0.745	0.588	0.807	0.765	0.618
GPT-4-Turbo-2024-04-09	0.696	0.677	0.520	0.729	0.694	0.535
Claude-3-Haiku-20240307	0.484	0.474	0.353	0.458	0.370	0.291
Claude-3-Opus-20240229	0.557	0.529	0.396	0.683	0.602	0.459
Claude-3-Sonnet-20240229	0.610	0.586	0.437	0.717	0.664	0.517

also presents a comparison to state-of-the-art methods; however, instead of traditional approaches, it focuses on deep learning-based methods. The methods are categorized into three groups: (i) approaches reimplemented in PyTorch 2.7.0 and evaluated using 100 random train-test splits, (ii) results extracted from original publications (potentially using different train-test split ratios), and (iii) multimodal LLMs evaluated in a zero-shot setting. Among the examined methods, MAMIQA [91] achieved the highest PLCC (0.895) and SROCC (0.874) on CLIVE, and MAMIQA also demonstrated the best performance on KonIQ-10k, with a PLCC of 0.937 and an SROCC of 0.926. Multimodal LLMs, including ChatGPT-4o-Latest, GPT-4o-2024-11-20, and Claude-3 series, were also evaluated. Among these, ChatGPT-4o-Latest performed best, reaching a PLCC of 0.758 and an SROCC of 0.732 on CLIVE, and a PLCC of 0.805 and an SROCC of 0.760 on KonIQ-10k, while these results are lower compared to specialized deep learning models, they indicate the potential of multimodal LLMs in IQA tasks.

5. Discussion

The results of this study demonstrate that multimodal LLMs exhibit varying levels of effectiveness in performing NR-IQA. The best-performing models, ChatGPT-4o-Latest and GPT-4o-2024-11-20, achieved superior correlation with human-annotated ground-truth scores, surpassing all traditional NR-IQA methods. This suggests that state-of-the-art LLMs are capable of capturing complex visual distortions in a manner that aligns well with human perception. However, Anthropic’s Claude series (Claude-3-Haiku, Claude-3-Opus, and Claude-3-Sonnet) provided results comparable to those of traditional NR-IQA algorithms, indicating that not all LLMs are equally effective in visual quality assessment. This discrepancy highlights differences in multimodal architectures and training methodologies between models developed by OpenAI and Anthropic. Despite the strong performance of the best LLMs, modern deep learning-based NR-IQA models still outperformed them. This suggests that specialized deep learning models, which are explicitly trained for IQA tasks with large-scale image datasets, retain an advantage over general-purpose multimodal LLMs. The limitations of LLMs in IQA may stem from their training focus on textual and general visual understanding rather than fine-grained image quality assessment. Additionally, some LLMs occasionally refused to provide ratings, particularly citing privacy concerns. These refusals were more frequent in the GPT-4o series, particularly for the CLIVE dataset, suggesting that model safety mechanisms influence the applicability of LLMs in IQA tasks. This raises an important consideration: while LLMs are capable of performing IQA, their reliability may be impacted by built-in safety constraints that prevent them from evaluating certain images.

6. Conclusions

This study presented a systematic evaluation of multimodal LLMs in NR-IQA. The findings indicate that state-of-the-art LLMs, particularly ChatGPT-4o-Latest and GPT-4o-2024-11-20, outperform all traditional NR-IQA methods but remain inferior to modern deep learning-based approaches. The Claude series models perform at a level similar to traditional NR-IQA methods, highlighting the variance in multimodal capabilities across different architectures. While LLMs offer promising performance in image quality evaluation, their occasional refusal to assess certain images suggests that their deployment in real-world NR-IQA applications requires careful consideration. Future work could explore fine-tuning LLMs specifically for IQA tasks or integrating them with deep learning-based approaches to enhance their robustness and accuracy.

A promising direction for future work is to integrate LLMs with traditional deep learning-based NR-IQA models. For instance, CNN-based models excel at capturing low-level distortions and fine-grained spatial artifacts, whereas LLMs (especially multimodal ones) can provide high-level semantic reasoning and linguistic interpretation of image quality. Combining these modalities, e.g., by using the CNN features as inputs to an LLM or as auxiliary inputs alongside language prompts, could lead to systems that not only predict quality scores but also generate natural language explanations aligned with human perception. Such hybrid models may benefit from both data-driven visual priors and the generalization capabilities of LLMs trained on large-scale multimodal datasets.

In future work, we plan to further investigate the role of open-source multimodal models in no-reference image quality assessment, while open-source models such as LLaVA and BLIP-2 have shown promise in various tasks, their performance in this domain currently lags behind closed-source models like those from OpenAI and Claude.AI. As open-source models continue to evolve, we anticipate that improvements in their architecture and training data will enhance their performance, making them more viable for tasks like image quality assessment. A more extensive comparison between closed-source and open-source models, including a discussion on the implications for performance, reproducibility, and accessibility, will be an important direction for future research. This will help to clarify the trade-offs between model openness and the quality of results, paving the way for more accessible and reproducible solutions in the field.