From Context to Human: A Review of VLM Contextualization in the Recognition of Human States in Visual Data

Abstract

This paper presents a narrative review of the contextualization and contribution offered by vision–language models (VLMs) for human-centric understanding in images. Starting from the correlation between humans and their context (background) and by incorporating VLM-generated embeddings into recognition architectures, recent solutions have advanced the recognition of human actions, the detection and classification of violent behavior, and inference of human emotions from body posture and facial expression. While powerful and general, VLMs may also introduce biases that can be reflected in the overall performance. Unlike prior reviews that focus on a single task or generic image captioning, this review jointly examines multiple human-centric problems in VLM-based approaches. The study begins by describing the key elements of VLMs (including architectural foundations, pre-training techniques, and cross-modal fusion strategies) and explains why they are suitable for contextualization. In addition to highlighting the improvements brought by VLMs, it critically discusses their limitations (including human-related biases) and presents a mathematical perspective and strategies for mitigating them. This review aims to consolidate the technical landscape of VLM-based contextualization for human state recognition and detection. It aims to serve as a foundational reference for researchers seeking to control the power of language-guided VLMs in recognizing human states correlated with contextual cues.

1. Introduction

We, as humans, have a human-centric perspective of the world. Consequently, many human-related aspects have become focal points of research in computer vision. These include action recognition (sometimes encompassing anomaly/violence detection) and human–computer interaction (which covers emotion recognition).

The most common approach in recent computer vision research is to define a benchmark consisting of a training set and a test set, develop a learning model on the training set, and evaluate it on the test set. However, in this basic form, the learning model cannot by itself overcome the limitations of the training data. A notable limitation is the finite amount of available data, which has led to the trend of introducing larger and richer datasets [1,2]. Another clear limitation is revealed when a model trained on one benchmark performs significantly worse when tested on another [3,4]. As a direct consequence, the research community has actively pursued methods to improve performance. One direction is to design methods specifically for cross-domain evaluation and comparison [5]. Another direction is to use machine learning tools trained on much larger datasets, thereby mitigating the limitations of the original benchmarks. The latter models are named foundation models or large models [6].

One approach involves incorporating an image captioning or caption-inspired language branch into the solution architecture, providing textual descriptions of visual inputs. Recall that image captioning is the process in which a (foundation) model analyzes an image, identifies objects, actions, and relationships, and generates a short natural language sentence or description explaining what is happening in the image [7]. A brief illustration of the captioning principle applied to human state recognition problems is shown in Figure 1. The key assumption underlying their success is that image captioning relies on VLMs, which are trained on datasets far larger than standard benchmarks. These models can provide richer context, allowing more nuanced predictions compared to a simple model that merely associates content with a label.

This narrative review surveys vision–language models (VLMs) for human-centric tasks. Key references are identified based on author expertise and citation chasing, and the literature is organized thematically, covering developments from the first concepts from 2019 to the emergence of the main research directions. From the beginning, VLMs were noted for their capacity to be used in downstream image tasks. First, Lu et al. [8] demonstrated that a single vision + language pre-trained model (though not yet large scale) could be adapted to a range of downstream vision–language tasks. Next, Radford et al. [9], introducing the CLIP paradigm, showed that a large-scale VLM can serve as a general zero-shot/few-shot solution for many computer vision problems (image classification, open-vocabulary recognition, etc.). More recently, Zhang et al. [10] highlighted that “Multi-Branch” architectures (vision encoder + text encoder) allow models to use natural language as a bridge for solving tasks without exhaustive per-task labeling, improving performance across diverse applications like autonomous driving [11], remote sensing [12], and robot vision [13]. Relating to the specific directions approached in this work, the use of VLMs has been demonstrated by Wang et al. [14] (the work became publicly available in 2021 as pre-print), thus initializing the use of VLM captioning/descriptions for human action recognition, and by Li et al. [15] (the work became public in 2023 as pre-print) for expression recognition.

As we will show later, there exist multiple cases in relation to human-centric aspects where the addition of an image captioning branch brought improvement. For instance, Li et al. [15] reported an increase in emotion recognition accuracy from 90.91 to 91.61, while Xenos et al. [16] reported an increase in EMOTIC from 29.19 to 34.66, achieved by addition of captioning. In violence detection, Wang et al. [17] reported a 4% increase in accuracy when adding text description. However, while greatly diminished compared to limited models, large models and their databases still have limitations and biases [18,19,20]. We will specifically review these while providing a mathematical formulation, thus further facilitating development in this direction.

Unlike prior surveys that focus on a single task or generic image captioning, this review jointly examines multiple human-centric problems within a unified contextualization framework. From an application perspective, we group the existing works into task categories. We separately discuss human action recognition and anomaly detection (including violence detection) and emotion recognition from facial cues, full-body posture, and context. Each category exhibits distinct challenges in terms of context reliance and bias sensitivity.

The three main contributions of this work are as follows: First, it provides a systematic review of VLM-based approaches for human-centric state recognition in visual data, covering human action recognition with outliers (e.g., violence detection) as well as emotion recognition from both full-body and facial expressions. While broader surveys of VLMs for visual recognition exist, this is, to the best of our knowledge, the first to focus specifically on these tasks, offering a consolidated categorization of existing work. Second, it proposes a unifying perspective of diverse research directions in human-centric state recognition. Third, in addition to highlighting the advances enabled by VLMs, it critically discusses their limitations, particularly human-related biases, and argues for the explicit integration of debiasing or bias mitigation mechanisms in future methods.

The remainder of the paper is organized as follows: Section 2 identifies summarizing reviews that focus on one of the directions approached in this paper. In Section 3, the prototypical solution is presented. Core contributors are the vision–language models (VLMs) which are detailed in Section 4. A factor affecting future potential success is the problem of bias in datasets and VLMs, which is investigated in Section 5. The second part of this paper focuses on narrow themes such as action recognition (Section 6), anomaly detection (Section 7), emotion recognition in context (Section 8), and facial expression recognition (Section 8). We end the paper with a discussion in Section 9 and conclusions in Section 10.

2. Related Work

This paper investigates the complex solutions with multiple branches that have emerged following the rise of VLMs. It spans several areas, with the specific aim of summarizing the impact that VLMs have on tasks related to humans in images and videos. Consequently, its purpose and scope overlap with additional research areas, and, in this section, we provide targeted references to help readers better understand the broader picture.

A key component of the solutions we analyze is the use of large models. These models first appeared in natural language processing as large language models (LLMs), and we refer readers to the work of Chang et al. [21] for a more thorough presentation. The concept has since been extended to VLMs, and the directions in architectures and tasks are summarized in the work of Zhang et al. [10]. In the reviewed literature, the approach often relies on prompts to guide and extract informative captions. A review focused on this direction is provided by Gu et al. [6]. The task may be considered an extension of image captioning, which has also recently been reviewed by Sharma et al. [7]. Given that, in most of the solutions we review, the VLMs are employed in a zero-shot learning framework, we point out that more details can be found in [22].

The popularity of VLMs in such solutions is largely due to their higher capacity and reduced overfitting and bias. A summary of the biases in large-scale image collections is provided in the works of Navigli et al. [23] and Sun et al. [24]. However, while discussing the bias in large models, the research community has tended to focus more on social biases. Key findings and developments have been summarized by Gallegos et al. [18]. The specific problem of bias in machine learning has been reviewed in several works, some of the more recent of which are those of Hort et al. [25] and Pagano et al. [26].

In the second part of this paper, we explore specific themes such as human action recognition, for which a comprehensive review is available in the work of Kong and Fu [27]. A subarea of this theme is violence detection, with recent developments presented by Mumtaz et al. [28].

Another direction addressed in this work is the recognition and exploration of human emotions as captured in images and videos [29]. This is a broad theme that encompasses several subdirections, including facial expression recognition from portrait images [30] and emotion recognition from context and body language [31].

Several others have explored multimodal or conventional image-based human state recognition without using vision–language models, including Hu et al. [32] for violence detection and Wang et al. [33] for video emotion recognition. While demonstrating the importance of the tasks, these methods do not leverage VLMs or caption-based embeddings and are therefore complementary to the approaches reviewed in this paper.

The works mentioned in this section focus on a specific theme, discussing in detail the problematic challenges and proposed solutions, where all architectures are considered. Existing reviews are largely fragmented, focusing on single tasks such as action or emotion recognition, while captioning surveys emphasize generation quality rather than the use of captions as contextual representations for downstream recognition. Furthermore, prior works rarely analyze how VLMs propagate bias into human-centric classifiers from a mathematical or architectural perspective, and they lack a unified comparison of how the same contextualization mechanisms perform across different human state recognition tasks. In our work, we suggest that the rise of VLMs leads to an encompassing solution which may be particularized to various directions but where the key element is the captioning of the input data. Thus, this work proposes a novel integrating perspective concentrated on the gains and disadvantages of incorporating VLMs into architectures dealing with human states in visual data.

3. General Architecture

Individual solutions differ in their approach and may be adapted to the theme, yet the use of image captioning to describe the context in which human-centric actions take place remains a mainstay.

A general architecture of human centric recognition that includes VLM captioning is presented in Figure 2. There, the input data depends on the problem; it can be either a single image or a video. In some cases, a preliminary pre-processing step is applied where the person or their face is detected, and only that region is used in subsequent processing.

With respect to the parallel branches, the following apply:

• Image description is marked with blue. The image content can be described using either a convolutional neural network (CNN, suitable for low-resource settings) or a Visual Transformer (ViT).
• Problem-specific descriptors. The central branch is optional and depends on the problem. For example, in action recognition or body emotion, it may include a system that identifies keypoints on human articulations (a skeletal representation). For facial expression recognition, it may involve facial keypoints or an action unit descriptor. For recognition in video, an optical flow descriptor may be used.
• VLM captioning. The branch highlighted in red is the focal point of this paper.

Explicit implementations of problem-specific descriptors will be presented in the respective task sections of the paper. In practice, in the pursuit of accuracy and efficiency, various solutions resort to different strategies to combine the three branches, such as early, sequential, and late fusion, depending on tasks requirements. These strategies will also be detailed in Section 6, Section 7 and Section 8 and concrete models and results will be presented.

Formalization

About VLMs, we note that often their use is prohibitive due to high resource demands. On the one hand, it is intuitive to construct the entire solution as a single large model (LM)—a general model carefully fine-tuned to balance generalization capability, optimize performance on the task of interest, and avoid memorization and overfitting [34]. In this case, if an input sample i is denoted by ${\mathbf{x}}_i\in \mathcal{X}$ with associated label $y_i$, the LM, represented by a function $f_{LM}$, is trained using a loss function $\mathcal{L}(y_i,\widehat{y_i})$. Here, $\widehat{y_i}$ is the prediction, obtained as follows:

$$\widehat{y_i}=f_{LM}\left({\mathbf{x}}_i\right).$$

(1)

The large model includes a prediction layer specifically designed to address the problem at hand.

Training an LM is resource-intensive, and only a limited number have the resources required [35]. Thus, most approaches prefer to employ an off-the-shelf VLM, use it to generate a textual description (caption) of the scene, and then convert the caption into a numerical vector (embedding). Formally, this can be described as follows:

$${\mathbf{e}}_{VLM}^i=f_{VLM}\left({\mathbf{x}}_i\right),$$

(2)

In this case, $f_{LM}$ is a cascade of a VLM and a Word2Vec model:

$$f_{LM}=f_{w2vec}\left(f_{VLM}(·)\right).$$

(3)

Other parallel branches may also be incorporated. These are typically trained (rarely from scratch, often fine-tuned) or used off-the-shelf to produce additional embeddings:

$$\begin{array}{cc}{\mathbf{e}}_1^i& =f_1\left({\mathbf{x}}_i\right),\hfill \\ \dots & =\dots ,\hfill \\ {\mathbf{e}}_p^i& =f_p\left({\mathbf{x}}_i\right).\hfill \end{array}$$

(4)

The concatenated embedding then serves as the representation of the data:

$${\mathbf{x}}_i\to [{\mathbf{e}}_1^i,\dots ,{\mathbf{e}}_p^i,{\mathbf{e}}_{VLM}^i].$$

(5)

Fusion may be performed directly, but it is preferable to train a smaller model on the concatenated embedding space to obtain the final prediction $\widehat{y_i}$.

4. Vision–Language Models

In this section, we present a short mathematical formulation of VLMs. According to Gallegos et al. [18], a large model (LM) parametrized by $\theta$ is defined as a model with an autoregressive, autoencoding, or encoder–decoder architecture trained on a corpus of hundreds of millions to near trillions of tokens. These LMs, with respect to the function, are language large models (LLMs, historically first), vision–language models (VLMs), or Large Multimodal Models.

From a constructive point of view, LMs are built on several paradigms that include autoregressive models such as GPT [36,37]; autoencoding models like BERT [38], RoBERTa [39], and XLM-R [40]; and encoder–decoder models such as BART [41].

Large models are commonly adapted for a specific task. LLMs address tasks such as text generation, sequence classification, or question answering, typically via fine-tuning. VLMs address vision problems such as image captioning and other downstream tasks (e.g., classification, object detection, etc.) [7,10]. Multimodal (general) models can handle multiple modalities beyond just vision or text, like text, images, audio, video, code, sensor data, etc. VLMs are specifically within the scope of this work.

Since VLMs are derived from LLMs, they can be divided into the following categories based on how the two modalities are fused: (i) early fusion models, which jointly encode an image and text from the start, such as UNITER [42], (ii) late fusion models, which have separate encoders for vision and language, then combine them, such as CLIP [9], (iii) unified multimodal Transformers, such as Flamingo [43] and BLIP-2 [44], and (iv) models extending pre-trained LLMs with vision, such as LiT [45] or LLava [46].

4.1. Categories of VLMs

A more relevant point of categorization is maybe the training paradigm, which leads the VLMs to be broadly categorized into four families [47]:

• Contrastive (dual-encoder) models;
• Generative/masked modeling VLMs;
• Pre-trained backbones with adapters;
• Unified multimodal Transformers.

4.1.1. Contrastive (Dual-Encoder) Models

Contrastive models make use of two distinct encoders—a visual encoder and a textual encoder—that project their respective inputs into a shared embedding space. For instance, CLIP [9] is trained with a contrastive loss to align paired images and texts. Such models are highly effective for tasks that can be reduced to similarity comparisons in the joint embedding space (e.g., zero-shot classification).

Training based on contrastive learning [47] follows the Energy-Based Models (EBM) paradigm [48], where a model $f_{\theta }$ is optimized to assign low energy to observed variables and high energy to unobserved ones. Data sampled from the target distribution should be given low energy, whereas out-of-distribution samples should receive higher energy. The corresponding Boltzmann distribution over input data $\mathbf{x}$ is expressed as follows:

$$p_{\theta }\left({\mathbf{x}}_i\right)={\displaystyle \frac{e^{-f_{\theta }\left({\mathbf{x}}_i\right)}}{{\sum }_{{\mathbf{x}}_i}{e}^{-f_{\theta }\left(\mathbf{x}\right)}}}$$

(6)

The maximum-likelihood objective models the underlying distribution $P_X$ from which inputs are drawn:

$$f_{\theta }=\underset{\theta }{argmin}{E}_{\mathbf{x}\sim P_X\left(\mathbf{x}\right)}[-log{p}_{\theta }\left(\mathbf{x}\right)]$$

(7)

However, evaluating $\mathbf{x}\sim p_{\theta }\left(\mathbf{x}\right)$ is generally intractable. To bypass it, Noise Contrastive Estimation (NCE) [49] reformulates the task as binary classification; the model predicts $C=1$ for real data samples and $C=0$ for samples from a noise distribution. Thus, the model learns to distinguish authentic data points from noise. The resulting objective is a binary cross-entropy loss:

$${\mathcal{L}}_{NCE}\left(f_{\theta }\right)=-\sum _{i}logP(C_i=1|{\mathbf{x}}_i;\theta )-\sum _{j}logP(C_j=1|{\mathbf{x}}_j;\theta )$$

(8)

with ${\mathbf{x}}_i$ drawn from the data distribution and ${\mathbf{x}}_j\sim p_n\left(\mathbf{x}\right),j\ne i$ drawn from the noise distribution.

In the original CLIP model [9], positive examples consist of an image paired with its ground-truth caption, while negative examples are constructed by pairing the same image with captions of other images from the mini-batch. A key innovation in CLIP is the training of vision and text encoders within a unified representation space. The encoders, initialized randomly, are optimized to produce similar embeddings for paired images and captions via a contrastive loss. Trained on 400 million image–caption pairs from the web, CLIP demonstrated striking zero-shot classification transfer performance.

CLIP-based VLMs exhibit stronger contextual and compositional capabilities than earlier or alternative frameworks due to their contrastive training on image–text pairs and the use of language as the sole supervision signal. By aligning entire images with entire sentences rather than individual regions or words, CLIP models learn concepts relationally (e.g., “a person riding a horse” versus “a person next to a horse”), making contextual information—such as attributes, relationships, and actions—central to their representations.

This is particularly advantageous for zero-shot tasks like violence detection, where labeled data for all categories is often unavailable and traditional supervised models struggle. CLIP-based models instead compare visual inputs directly with textual descriptions (e.g., “a person hitting another person” or “a crowd running violently”), and can similarly model emotions in context by considering the full image rather than relying solely on facial cues.

4.1.2. Generative/Masked Modeling VLMs

Generative approaches, exemplified by BLIP [50], are typically trained on caption generation or masked language modeling objectives. They can be understood as a variant of denoising autoencoders [51], where the noise follows a spatial pattern. This perspective also connects them to image inpainting methods, such as those proposed by Pathak et al. [52], which exploit missing regions to learn rich visual representations.

From a theoretical standpoint, Dubois et al. [53] demonstrate that any transformation $f\left(\mathbf{x}\right)$ applied to data $\mathbf{x}$ implicitly induces an equivalence relation, partitioning $f\left(\mathbf{x}\right)$ into disjoint equivalence classes. Conditional densities remain constant within a class, i.e., $f\left(\mathbf{x}\right)\sim f\left({\mathbf{x}}^{\prime }\right)⟶p\left(\mathbf{z}\right|f\left(\mathbf{x}\right))=p\left(\mathbf{z}\right|f\left({\mathbf{x}}^{\prime }\right))$, where Z represents the learned representation of X. This perspective unifies masking, augmentation, and modality selection as transformations of data. The problem can then be related to a rate–distortion formulation [54] as follows:

$$\underset{p\left(\mathbf{z}\right|\mathbf{x})}{argmin}I(f\left(\mathbf{x}\right);Z)+\beta ·H\left(X\right|Z)$$

(9)

Recovering the masked VLM objective involves bounding the above expression, yielding the following:

$$\mathcal{L}=-\sum _{x\in \mathcal{X}}{E}_{p\left(f\right)p\left(Z\right|f\left(\mathbf{x}\right))}[logq\left(\mathbf{z}\right)+\beta ·logq\left(\mathbf{x}\right|\mathbf{z})].$$

(10)

Here, $logq\left(\mathbf{z}\right)$ serves as an entropy bottleneck, limiting the rate $I\left(f\right(\mathbf{x});Z)$ by filtering redundant information. In masking-based VLMs, this entropy bottleneck is often bounded by a constant proportional to the information removed during masking. For multimodal VLMs, Z is constrained to retain the minimum essential information across modalities. Meanwhile, $logq\left(\mathbf{x}\right|\mathbf{z})$ bounds the distortion $H\left(Z\right|X)$, ensuring that predictive information is preserved. In practice, this term corresponds to autoencoding.

Unlike CLIP-style models that score hypotheses, generative VLMs aim to explain the scene. Trained to predict missing words conditioned on visual input, they are more sensitive to verbs, verb–object relations, and prepositions, which is valuable when linguistic reasoning, temporal abstraction, or label scarcity is important. For example, completing “A person is [MASK] toward a crouched man” requires inferring body pose and motion direction—key cues for tasks such as violence detection or emotion recognition from body posture. Therefore, they are advised when scene details matter more.

4.1.3. Pre-Trained Backbones with Adapters

A third family of approaches augments large frozen backbones with lightweight adapter modules. Models such as BLIP-2 [44], LLaVA [46], and Flamingo [43] adopt the following strategy: the vision encoder or large language model is kept frozen, while compact adapters, such as the Q-Former, are trained to connect the modalities, yielding a fraction of the full training cost. We recall that Q-Former is a lightweight Transformer that uses learnable query vectors to extract key visual features from a frozen image encoder. It selects the most relevant visual information to feed into a frozen LLM, enabling the model to generate the desired text efficiently.

This category of VLMs is more useful when reasoning, temporal aggregation, or task adaptation matters more than raw visual matching. In this setup, the model’s strength lies in the language prior, so visual features must be interpreted in context rather than simply embedded. From an accuracy perspective, these models are recommended when actions are ambiguous, without intent or causality—for example, distinguishing “pushing” from “helping” or “intimidated by a crowd” from “angry toward the crowd”. They are particularly effective when fine-tuned data is scarce, or when recognition relies on frame-level data and the sequence must be analyzed at the frame level.

4.1.4. Unified Multimodal Transformers

Unified multimodal Transformers such as Flamingo [43] or Kosmos-2 [55] process textual and visual inputs jointly within a single model. Typically, this is achieved via early fusion, where both images and text are tokenized and jointly attended to through co-attention or a shared embedding space.

Unified VLM models are task-agnostic, performing equally well—or equally poorly—across different tasks. In general, they excel at fluid generative tasks and complex cross-modal reasoning. For example, they can infer “What action is most likely occurring and why?”, since the same model parses the image, reasons about actions, and generates an answer or label. On the other hand, while these models can handle heterogeneous actions, they are computationally demanding and require large, carefully curated datasets.

4.2. Practical Aspects

4.2.1. Choosing a Model

The optimal choice of VLM depends on the downstream task [47]. Retrieval and matching tasks are best served by contrastive models, while generation-oriented tasks such as captioning or Visual Question Answering (VQA) benefit more from generative or unified architectures. Adapter-based methods strike a practical middle ground between efficiency and flexibility, making them attractive for resource-constrained environments or when reusing powerful frozen backbones.

4.2.2. Dataset Sizes and Curation

A defining aspect of VLMs is their reliance on very large-scale datasets, often spanning hundreds of millions to billions of image–text pairs. Dataset scale plays a critical role in enabling broad generalization across tasks and modalities. At the same time, dataset quality is equally vital. Raw web data is noisy by default, requiring filtering and pre-processing to yield reliable alignment. In many cases, the training data is not publicly released, though accompanying reports often describe the collection strategy, which typically includes the following:

• Web-scale collections of image–text pairs;
• Filtering and bootstrapping for noise reduction;
• Curated domain-specific datasets.

For example, BLIP employs caption bootstrapping to improve noisy web data, whereas curated datasets are crucial for fine-tuning domain-specific applications. To systematically evaluate dataset quality, DataComp [56] introduces a benchmark in which the CLIP architecture and training hyperparameters are fixed, isolating the impact of the pre-training dataset. Results show that dataset diversity and domain alignment significantly affect downstream performance and robustness. More broadly, scaling laws observed in large neural networks [57] emphasize that both dataset size and careful curation are central to advancing multimodal learning.

Fine-Tuning Strategies

Because training VLMs from scratch is highly resource-intensive, fine-tuning strategies have been adapted and several popular methods exist [47]:

• Full fine-tuning of all model parameters;
• Adapter-based fine-tuning (e.g., LoRA);
• Prompt tuning;
• Instruction tuning;
• Probing classifiers.

Full fine-tuning, where all parameters are updated, is generally the most effective but also the most computationally expensive. Adapter-based techniques such as Low-Rank Adaptation (LoRA) [58] insert trainable modules into frozen backbones, allowing efficient adaptation with minimal parameter updates. We recall that LoRA (Low-Rank Adaptation) efficiently fine-tunes large VLMs by updating only small, low-rank matrices in certain layers, keeping most of the original model frozen and thus using much less computation and memory. Prompt tuning [59] goes even further, learning task-specific soft prompts—textual or visual—without modifying backbone weights.

Instruction tuning [46] has proven especially powerful in models like LLaVA and InstructBLIP, where training on instruction-style input–output pairs enhances zero-shot reasoning and alignment with human intent. Probing classifiers [60] offer another lightweight approach: embeddings from frozen VLMs can be used to train compact classifiers for downstream tasks such as multimodal fact-checking.

5. Bias in VLMs and Implications

Bias in predictors (including VLMs) usually refers to the difference between the expected prediction of the model and the true value it is trying to estimate. Mainly, the bias quantifies a systematic error introduced by the model prediction [61]. Two root causes are often identified as the source of the model bias: training dataset bias and model assumptions-derived bias. Dataset bias refers to systematic errors or distortions in the data that cause certain patterns, groups, or outcomes to be over- or under-represented. This means the dataset does not fully or fairly reflect the real-world distribution.

Probably the first and one of the most influential arguments is the “Name That Dataset” experiment introduced by Torralba and Efros [62]. There, each dataset is treated as a class in a classification task, and models are trained to predict the dataset of origin for each image. As early as 2011, SVMs were able to classify datasets with high accuracy [62], which highlights the built-in bias of visual datasets. More recently, Liu and He [63] revisited this experiment and reported 84.7% accuracy on a three-way classification task involving the YFCC, CC, and DataComp datasets. Furthermore, increasing the number of training samples or using stronger data augmentation further improved accuracy on held-out validation data even though the classification task itself became more challenging.

A key observation is that these datasets should be difficult to distinguish from one another; this separability indicates strong built-in dataset bias, and later studies sought to identify its sources. For example, Zeng et al. [19] analyzed object-level semantics and used language models to describe dataset-specific traits.

Concerning model bias, You et al. [64] showed that classifiers can easily distinguish images generated by different diffusion model families, though not within the same family. This implies that model-specific assumptions shape outputs in ways imperceptible to humans, demonstrating that unobservable biases may still meaningfully exist.

With respect to bias analysis and identification, there are two main directions: The first direction is more tangible, focusing on identifying social bias in both large databases and large models. This direction relies more heavily on basic statistical measurements that reveal non-uniformity and under- or over-representation of specific categories based on easily identifiable human traits such as age, gender, or race. The second direction is more algorithmic and aims to detect and isolate biases through the analysis of databases and, correspondingly, the models. Often, the outcome is the identification of a category, but this category does not necessarily have a clear real-world interpretation. In the next subsection, we will review key works and analyze both directions after formalizing the mathematical notations.

5.1. Mathematics of Bias

Inductive Bias

Let us now formalize the notion of bias. We begin by discussing the problem in inductive learning, where the goal is to use a dataset $({\mathbf{x}}_i,y_i),i=1\dots N$ to compute a model function $f_{\theta }\left(\mathbf{x}\right)$ such that $f_{\theta }\left({\mathbf{x}}_i\right)$ approximates $y_i$ accurately. Here, ${\mathbf{x}}_i\in \mathcal{X}$ denotes a data instance, while $y_i\in \mathcal{Y}$ represents the corresponding output, target variable, or label. The search for $f_{\theta }$ is carried out within a specific set $\Theta$, and the preference for this set is referred to as bias by Mitchell [65].

The inductive learning process is formulated as the following minimization problem:

$$f_{\theta }=\underset{\theta }{argmin}\mathcal{L}\left(f\right),$$

(11)

where training corresponds to solving the minimization in the equation (Equation 11). The training set $({\mathbf{x}}_i,y_i),i=1\dots N$ is used to identify the parametric function $f_{\theta }$ that minimizes the loss function $\mathcal{L}$.

All machine learning techniques for inductive learning require some form of inductive bias to function effectively, and the choice of $\Theta$ is often a critical design decision. If the inductive bias is too low (i.e., $\Theta$ is too large), the model may overfit, allowing noise in the data to unduly influence the choice of $f_{\theta }$. This results in poor generalization, meaning that $f_{\theta }$ fails to approximate unseen data. Conversely, if the inductive bias is too high (i.e., $\Theta$ is too small), the model may underfit, producing poor approximations. Thus, inductive bias is essential, but it must be applied in the right amount.

5.2. Dataset Bias

As mentioned, bias in datasets refers to systematic errors or distortions that cause certain groups to be over-represented or unnaturally correlated with specific patterns. Following the approach of Gallegos et al. [18], we define the groups as follows:

A group $G\in \mathcal{X}$ is a subset of the data that shares a particular trait, which may be fixed, contextual, or socially constructed. For example, if $\mathcal{X}$ describes persons (e.g., portrait images), the shared trait may correspond to, e.g., age, skin color, disability, gender identity, national origin, or race.

In the more general case of images, such traits may be less intuitive to define. For instance, a group might consist of images containing an unusually high concentration of high-frequency components (e.g., dominated by very small objects) or images where textures predominantly align within a specific range of orientations.

For a model to be considered unbiased, it must perform fairly across all possible groups. To formalize group fairness, we consider a model $f_{\theta }$ producing outcomes $\widehat{Y}=f_{\theta }(X;\theta )$. Given a set of groups $\mathcal{G}$, group fairness requires approximate parity (within a tolerance $ϵ$) across all groups $G\in \mathcal{G}$ with respect to a statistical outcome measure $M_Y\left(G\right)$ conditioned on group membership:

$$|M_Y\left(G\right)-M_Y\left(G^{\prime }\right)|\le ϵ,\forall G,G^{\prime }\in \mathcal{G}.$$

(12)

The choice of M specifies the fairness constraint, which is inherently subjective and context-dependent. Examples of M include accuracy, true-positive rate, and false-positive rate.

Group fairness may be affected by unwanted correlations [61,66]. To quantify these effects, a bias score b for an outcome $\widehat{y}$ with respect to a group G is calculated as follows:

$$b(\widehat{y};{\mathbf{x}}_g)={\displaystyle \frac{c(\widehat{y};{\mathbf{x}}_g)}{{\sum }_{{\mathbf{x}}^{\prime }\in G}c(o;{\mathbf{x}}^{\prime })}},$$

(13)

where $c(o;{\mathbf{x}}_g)$ denotes the number of co-occurrences of o and ${\mathbf{x}}_g$ within group $\mathbf{x}\in G$. If $b(\widehat{y};{\mathbf{x}}_g)>\frac{1}{\left|G\right|}$, then o is positively correlated with ${\mathbf{x}}_g$, indicating that the data is biased with respect to this group. Lately, this has been referred to as spurious correlation [67].

A stricter requirement is individual fairness, which demands fairness at the level of each individual data point. Considering [18], let $\mathbf{x},{\mathbf{x}}^{\prime }\in X$ be two individuals, and let $d:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ denote a distance metric. Let O be the set of possible outcomes, and let $f_{{\theta }_M}:\mathcal{X}\to \Delta \left(O\right)$ denote a mapping from an individual to a distribution over outcomes. Individual fairness requires that individuals who are similar with respect to the task receive similar treatment, i.e.,

$$\forall \mathbf{x},{\mathbf{x}}^{\prime }\in \mathcal{X}:D\left(f_{{\theta }_M}\left(\mathbf{x}\right),f_{{\theta }_M}\left({\mathbf{x}}^{\prime }\right)\right)\le d(\mathbf{x},{\mathbf{x}}^{\prime }),$$

(14)

where D is a divergence measure between distributions, such as statistical distance.

The trait defining a group may be either explicit and intuitive or subtle and harder to detect. Gallegos et al. [18] distinguish between these cases via the concept of fairness through unawareness. A model satisfies it if a sensitive group G is not explicitly used, such that

$$f_{\theta }\left(G\right)=f_{\theta }(\neg G)$$

(15)

where $\neg G$ denotes the complement of G, i.e., $\neg G=\mathcal{X}\setminus G$.

A model is further said to satisfy invariance if, for any two groups $G_i,G_j$, the outcomes $f_{\theta }\left(G_i\right)$ and $f_{\theta }\left(G_j\right)$ are identical under some invariance metric $\psi$.

Systematic error or bias is easier to measure when the trait of interest corresponds directly to a specific label value (in classification) or range (in regression). For example, in binary classification, one may require that the overall misclassification rate (OMR) be independent of a protected attribute A (where $A\in G_i,G_j$). The corresponding fairness condition for the classifier is as follows [68]:

$$P(\widehat{Y}\mid A\in G_i,y=0)=P(\widehat{Y}\mid A\in G_i,y=1),$$

(16)

where $\widehat{Y}$ is the classifier output $f_{\theta }\left(\mathbf{x}\right)$, and y is the true label for input $\mathbf{x}$. Both $\widehat{Y}$ and y take values in $0,1$.

In Equation (16), the chosen measure is the OMR, but the property can equivalently be formulated using other quality metrics, such as the false-positive rate (FPR), false-negative rate (FNR), false omission rate (FOR), or false discovery rate (FDR) [61].

To enforce fairness, the model optimization process described in Equation (11) is constrained to ensure comparable error rates across all relevant groups $G_i\in \mathcal{G}$:

$$f_{\theta }=\underset{\theta }{argmin}\mathcal{L}\left(f\right),s.t.P(f_{\theta }\left(\mathbf{x}\right)\ne y\mid A\in G_i)=P(f_{\theta }\left(\mathbf{x}\right)\ne y\mid A\in G_j).$$

(17)

5.3. Theoretical Limitations in Bias and Fairness

Kleinberg et al. [69] rigorously examined the formulation in Equation (17) in relation to the binary classification case ($y\in \{0,1\}$) and introduced three fairness conditions: (A) calibration within groups (each bin in the database should proportionally reflect the true values across groups), (B) balance for positives, and (C) balance for negatives. Conditions (B) and (C) ensure that score assignments are not systematically more inaccurate for positive or negative instances in one group than in another. They established and proved the following theorem [69]:

Theorem 1.

Consider an instance of the problem in which there exists a risk assignment (or loss function) satisfying fairness conditions (A), (B), and (C). Then, the instance must either permit perfect prediction (with $f_{\theta }\left(\mathbf{x}\right)$ equal to 0 or 1 for all $\mathbf{x}$) or exhibit equal base rates.

This theorem can also be formulated and proven in terms of asymptotic tendencies. It primarily addresses formal model-level fairness by highlighting the trade-offs between dataset statistics (e.g., unequal base rates) and predictor capabilities. The intuition is straightforward: if a dataset has unequal base rates, then achieving fairness metrics requires predictor adjustments through prior assumptions that artificially distort the feature space. Such distortions, however, prevent perfect performance, especially when the test set approximates the true (but unavailable) population distribution.

These foundational results were later extended. Zhao and Gordon [70] derived a lower bound on the joint error of classifiers satisfying statistical parity under unequal base rates, demonstrating that fairness constraints necessarily induce certain error behaviors. Their work links the learning of fair representations with information-theoretic bounds, bridging predictor bias (how fairness constraints affect prediction error) and dataset bias (group-specific base rates and distributions). However, when the population contains more than two groups, the performance lower bound no longer has a closed-form analytical expression [70].

While these theoretical insights are valuable, their existence alone does not resolve the problem of bias. Practical interventions have been proposed, most notably by Hardt et al. [71], who modified classifiers to satisfy Equalized Odds and Equality of Opportunity. Given an unfair classifier $f_{\theta }$, they constructed a new classifier $f_{{\theta }_2}$ by solving an optimization problem with fairness loss terms. However, Woodworth et al. [72] later showed that such modifications can significantly reduce accuracy—for example, when the loss function is not strictly convex. Moreover, these methods require prior knowledge of dataset biases, such as group-specific label distributions.

Further work by Lee and Shou [73] investigated bias and its compensation in continuous outcomes. They demonstrated both theoretically and empirically that ML models often exhibit systematic biases in regression settings—for example, under-predicting large values and over-predicting small ones, a phenomenon known as central-tendency bias.

In the following subsections, we review recent and effective methods for mitigating bias. Since this work focuses on human-centric data, we also consider the relevance of social unfairness and examine studies that identify social biases in large models.

5.4. Bias Quantification and Mitigation

Bias is an aspect that still troubles the machine learning community. Over time, stronger and newer models become less prone to bias, but this problem still exists. To illustrate this, Figure 3 presents the prompt and images generated with an ML solution; the results show clear differences based on race between the two categories. On the other hand, different VLMs can exhibit subtle biases when generating descriptions of people (see Figure 4).

While surveying more than 100 studies on fairness, Hort et al. [25] identified several categories of bias mitigation methods, which first are classified depending on the moment of intervention. Afterwards, each category, is further detailed.

• Pre-Processing Methods:
  • Relabeling and Perturbation (RP): Modify training data either by changing ground-truth labels (relabeling) or altering feature values (perturbation).
  • Sampling (Samp): Adjust the training data distribution by adding or removing samples, or by reweighting their influence during training.
  • Latent Variables(LV): Augment training data with additional, preferably unbiased, features.
  • Representation Learning (Repr): Learn transformations of training data that reduce bias while retaining as much useful information as possible.
• In-Processing Methods:
  • Regularization and Constraints (RC): Modify the learning algorithm’s loss function. Regularization introduces penalty terms for discrimination (increasing loss when discrimination occurs), while constraints impose bias limits that cannot be violated during training.
  • Adversarial Learning (AvL): Train models alongside adversaries. The model predicts ground-truth values, while the adversary attempts to exploit fairness violations.
  • Compositional Approaches (CA): Train multiple classifiers, each specialized for a specific group (e.g., privileged vs. unprivileged), and combine their predictions.
  • Adjusted Learning (AdL): Adapt existing algorithms or develop new ones with explicit bias mitigation mechanisms.
• Post-Processing Methods:
  • Input Correction: Apply modifications to test data before prediction.
  • Classifier Correction: Adjust trained models directly to satisfy fairness criteria.
  • Output Correction: Modify predicted labels as a final step in enforcing fairness.

Bias can affect any of the ML modules that are used in the general architecture in Figure 2, and detailed discussions are provided in [25,26]. In this work, however, our primary interest lies in how biases within VLMs influence the final solution, and thus we focus on recent proposals that aimed to isolate and measure bias in vision–language models. A compact view of these solutions is given in Table 1.

Table 1. General bias estimation solutions for VLMs. Methods listed here aim to estimate and correct the bias of a VLM with respect to a specified group. The “Moment” refers to the point of intervention in the task processing, which may be before (“Pre-Proc”), during (“In-Proc”), or after, by correcting the prediction (“Post-Proc”).

Solution	Moment	Type	VLM Targeted	Benchmarks Reported
Revise [74]	Pre-Proc	RP, Samp	All	COCO; Places; Visual Genome, etc.
VLBiasBench [75]	Pre-Proc	Samp	All	Synthetic
Multifair [76]	Pre-Proc	Samp, Repr	All	CelebA
Zhu et al. [77]	Pre-Proc	RP	All	Debiased COCO; other
BiMa [78]	Pre-Proc	LV	All txt-vid retrieval	MSR-VTT; ActivityNet; etc.
OxonFair [79]	Pre-Proc	AdL	All	CelebA; others
DomInd [80]	In-Proc	RC	All	CIFAR-10S; CelebA
IDR [81]	In-Proc	RC	All	MiniImageNet; CalTech 101, etc.
BA-LoRA [82]	In-Proc	RC	All (LLM)	Waterbirds; CelebA
PRISM [67]	In-Proc	RC	CLIP	Waterbirds; CelebA
LogicCLIP [83]	In-Proc	RC	All	LogicBench
DIM [84]	In-Proc	CA, RC	All	CIFAR-100; Breeds
REAL [85]	Post-Proc	CC	All	ImageNet; Flowers; EuroSAT; etc
TriProTesting [24]	Post-Proc	CC	All	CelebA; UTKFace; FairFace, etc.

Table 1. General bias estimation solutions for VLMs. Methods listed here aim to estimate and correct the bias of a VLM with respect to a specified group. The “Moment” refers to the point of intervention in the task processing, which may be before (“Pre-Proc”), during (“In-Proc”), or after, by correcting the prediction (“Post-Proc”).

Solution	Moment	Type	VLM Targeted	Benchmarks Reported
Revise [74]	Pre-Proc	RP, Samp	All	COCO; Places; Visual Genome, etc.
VLBiasBench [75]	Pre-Proc	Samp	All	Synthetic
Multifair [76]	Pre-Proc	Samp, Repr	All	CelebA
Zhu et al. [77]	Pre-Proc	RP	All	Debiased COCO; other
BiMa [78]	Pre-Proc	LV	All txt-vid retrieval	MSR-VTT; ActivityNet; etc.
OxonFair [79]	Pre-Proc	AdL	All	CelebA; others
DomInd [80]	In-Proc	RC	All	CIFAR-10S; CelebA
IDR [81]	In-Proc	RC	All	MiniImageNet; CalTech 101, etc.
BA-LoRA [82]	In-Proc	RC	All (LLM)	Waterbirds; CelebA
PRISM [67]	In-Proc	RC	CLIP	Waterbirds; CelebA
LogicCLIP [83]	In-Proc	RC	All	LogicBench
DIM [84]	In-Proc	CA, RC	All	CIFAR-100; Breeds
REAL [85]	Post-Proc	CC	All	ImageNet; Flowers; EuroSAT; etc
TriProTesting [24]	Post-Proc	CC	All	CelebA; UTKFace; FairFace, etc.

Among the recent works that focus on large databases, the study of Wang et al. [74] presents REVISE, a tool that detects and mitigates dataset biases across objects, demographics, and geography early in the ML pipeline. Wang et al. [75] introduce VLBiasBench, a large benchmark evaluating social and intersectional biases in VLMs, finding commercial models to be generally less biased than open-source ones. Delaney et al. [79] present OxonFair, a customizable fairness toolkit that uses per-group thresholding to balance accuracy and fairness across domains. Tian et al. [76] introduce MultiFair, a mixup-based method that enforces fairness across multiple sensitive attributes simultaneously. Zhu et al. [77] reveal flaws in membership inference benchmarks for VLM auditing and construct unbiased datasets to show realistic auditing challenges.

With respect to bias mitigation methods that adjust the learning in-process, Wang et al. [80] created CIFAR-10S to benchmark debiasing strategies and showed that domain-independent training outperforms adversarial approaches in reducing spurious correlations. Zhang et al. [84] introduce DIM, a framework that discovers multiple biased subgroups in image classifiers and mitigates them for greater robustness. Ma et al. [81] propose Intrinsic Dimension Regularization, showing that balancing class perceptual manifold complexity improves fairness. Chang et al. [82] propose BA-LoRA, a fine-tuning method with novel regularizers that efficiently mitigates pre-training bias in LLMs. Molahasani et al. [67] introduce PRISM, a data-free, task-agnostic method that removes spurious bias in vision–language models using LLM-generated correlations and projection learning. Zhou et al. [83] introduce LogicBench and LogicCLIP, showing VLMs’ logical blindspots and enhancing reasoning with logic-aware training. Le et al. [78] propose BiMa, a framework that disentangles visual and textual biases in text–video retrieval to improve generalization.

Recent post-processing methods include REAL, a retrieval-augmented approach proposed by Parashar et al. [85] that uses LLMs to improve VLM performance on rare concepts in long-tailed distributions. Also, Sun et al. [24] proposed a two-stage method, first using the so-called TriProTesting to identify the point of unbalance and then, to mitigate it, using adaptive logit adjustment. Their work again emphasizes that VLMs conserve existing societal biases from the large databases that they have been trained on.

Overall, the methods summarized in this subsection are general, seeking numerical or statistical bias rather than a very specific and narrow category. Overall, some aspects must be emphasized: (i) Methods isolate bias, which is sometimes explainable, sometimes not, and emphasize that it exists. (ii) Zhu et al. [77], who evaluated multiple models, found them rather ineffective, and thus strongly concluded that there was a need for improvement. Methods that target particular, tangible biases, such as the so-called ethical and social biases, tend to be more effective, as directed algorithms show greater improvement. (iii) Biases and uneven performance may appear across groups, but, first, one needs to define the groups of interest. These are problem-dependent, and examples may exist outside societal biases—for instance, spurious correlations between certain background colors and corresponding emotions.

5.5. Social Bias and Fairness

Since the purpose of this paper is to examine human-centric solutions, it is logical to study the unevenness found in databases and foundational models that are likely transferable to specific solutions. The work of Gallegos et al. [18] integrates reviews, yet their work is focused on language models, and, while the results are partially transferable to visual models, they are not identical. With respect to VLMs, several directions can be identified in recent works.

5.5.1. Bias in Training Data and Representation Spaces

One perspective is that the root cause is in the large databases used for training. One particular problem is that, in many cases, such databases are not public and the bias can only be inferred. However, there is one public very large database. Here, Birhane et al. [86] critically examined LAION-400M, revealing that large-scale internet-scraped datasets contain pornography, slurs, and stereotypes, and they argue that “scale beats noise” is an inadequate justification for dataset quality.

5.5.2. Bias in Image Captioning Outputs

Works in this area separate the bias in the outputs of predictors. Zhao et al. [87] analyzed racial bias in image captioning, showing that darker-skinned individuals are described with less detail, lower accuracy, and differing sentiment compared to lighter-skinned individuals. Sabir et al. [88] investigated spurious gender correlations in captions, introducing a gender score to quantify how objects like “lipstick” are strongly and stereotypically linked to women. Abdelrahman et al. [89] introduced the ImageCaptioner metric to assess how captioning models amplify dataset bias for protected attributes, finding widespread and nontrivial bias amplification across architectures and datasets. Fraser et al. [90] constructed a parallel image dataset to demonstrate systematic differences in generated captions depending only on perceived gender or race, revealing stereotypical tendencies in LVLM outputs. Konavoor et al. [91] systematically mapped gender associations in contrastive vision–language encoders by comparing embeddings of male/female faces with labor-related statements, producing category-wise bias profiles with uncertainty estimates.

5.5.3. Mitigation-Oriented Approaches in Captioning

Here, works aim to both isolate bias and propose specific methods to mitigate it. Yang et al. [92] proposed adversarial masking to suppress gendered words in captions, replacing over 99% of gender markers with neutral terms while maintaining high captioning quality. Hirota et al. [93] developed LIBRA, a debiasing framework that addresses both contextual and stereotypical gender bias in captions, reducing misclassification while improving output fairness. Hamideh et al. [20] proposed the So-B-IT taxonomy of social biases and empirically show that CLIP associates harmful stereotypes (e.g., linking “terrorist” disproportionately with Middle Eastern men), with debiasing efforts often redistributing bias rather than eliminating it.

5.5.4. Normative and Philosophical Considerations

Other works take a more detached perspective. Buijsman [94] argues that fairness metrics in AI cannot all be optimized simultaneously and proposes Rawls’ justice as fairness as a guiding principle, prioritizing the least advantaged when navigating technical fairness–accuracy trade-offs.

In conclusion, these works collectively reveal that VLMs and captioning systems encode, amplify, and sometimes exacerbate social stereotypes along dimensions such as race, gender, and occupation, with biases arising both from pre-training datasets and model architectures.

Many studies in image captioning focus on gender, especially gendered pronouns, misclassification, and stereotypical associations. There are also solid works, though fewer, focusing on other axes like race, skin tone, intersectional bias, ableism, etc. Datasets used, like MSCOCO, often have their own bias, and quantification depends strongly on the annotation quality (e.g., perceived skin color, manual labeling). Some metrics treat any gendered terms as bad, which may not always be desired depending on context (if gender is visible and relevant). Therefore, distinguishing unnecessary or stereotyped gender mentions from correct visible gender mentions is a double-edged sword.

While debiasing strategies (e.g., masking, adversarial interventions, fairness-aware training) show partial success, bias removal remains an open challenge—feasible in controlled cases, but far from a solved problem, often trading one form of bias for another.

6. VLMs in Human Action Recognition

In this section, we explore recent progress in human action recognition (HAR), focusing on approaches that integrate multimodal reasoning and contextual understanding through VLMs. In the context of HAR, the video modality plays a central role, which implies the need for well-synchronized captions and, optionally, additional scene descriptors such as object detections. The domain is mature and the recent search focused on the specific benchmarks which are summarized in Table 2.

The idea is as follows: Modern VLMs built on large pre-trained language models can effectively leverage richer and more structured annotations, including scene context (e.g., “inside a kitchen”, “outdoors on a baseball field”, as in Kinetics-600 [95] or Ego4D [96]), object relations (e.g., “placing the cup on the table next to the laptop”, as in Something-Something V2 [97]), and goal-driven narratives (e.g., “searching for keys before leaving the house”, as in Ego4D [96] or YouCook2 [98]). Here, the term “fine-grained” refers to the ability to distinguish between actions that share similar motion patterns but differ in object identity, spatial relations, or intent—for example, reaching for versus grasping an object, or placing versus adjusting it—rather than assigning broad categories such as cooking or playing sports. These detailed annotations help resolve ambiguity between visually similar motions by providing richer semantic cues that ground entities and their interactions within a specific scene, enabling more precise action understanding.

In the research community, there is broad recognition that both scene context and object interaction relations are valuable for action recognition, and that the most effective models integrate them both rather than prioritizing one exclusively.

As outlined in the introduction, the main architectural approaches in VLMs can be grouped into four primary categories. Table 3 condenses the reviewed methods and highlights their advantages over unimodal approaches.

Table 2. Popular benchmarks for action recognition. In the observation columns, we nominate which context contributes most to HAR performance: Low—L, Medium—M, High—H.

Benchmark	Size/Duration	Data	Observations
MSR-VTT [99]	10k YouTube videos; 200k descriptions (20 captions per clip); 40 h	Videos (10–30 s each) + manually labeled captions (avg 10 words/caption)	Standard benchmark for video → text retrieval and video question answering (Video QA). Scene context: H (diverse environments, activities) + interaction context: L (implicit, coarse) → tasks: video–text retrieval, Video QA, captioning.
Youcook2 [98]	2k videos; 176 h (avg 5.26 min/video—each video is divided in 7–16 clips)	Cooking instructional videos + one caption per clip (10–20 words)	Used in instructional video understanding tasks (captioning, retrieval). Scene context: M (kitchen, cooking setup) + interaction context: M (human–object, procedural) → tasks: instructional captioning, retrieval, procedure understanding.
Kinetics-600 [95]	474k YouTube videos (avg 10 s/video); 1317 h	Videos + 600 distinct actions classes (e.g., sports, domestic tasks)	Mainly used for video-level action classification. Scene context: H (sports, indoor/outdoor cues) + interaction context: L (coarse labels) → tasks: action classification, representation, pre-training.
Ego4D [96]	3670 h collected from 923 participants across 74 locations and 9 countries (avg 8 min/video)	Egocentric videos (first-person view) + narrations—each narration is a free-form sentence associated with a single timestamp.	Scene context: H (egocentric, environment-aware) + interaction context: H (who–does–what–why, long-term)→ tasks: long-term action anticipation, temporal grounding, QA, NL retrieval.
Something-Something-V2 (SSv2) [97]	170K videos (220k clips with 4–5 s/clip); 276 h	Clips + labels—each clip is labeled with a template caption such as “Putting [something] onto [something else]”.	Widely used for action recognition tasks—specifically, video-level classification of fine-grained human–object interactions (e.g., “Pushing [something] from left to right”). Scene context: L (background suppressed) + interaction context: H (fine-grained object relations) → tasks: fine-grained, action recognition, interaction-centric classification.
COIN [100]	11k videos; 476 h; 46,354 annotated segments	Instructional videos, covering 180 distinct tasks across 12 different domains (e.g., vehicles, gadgets, household items) + annotations	It supports tasks such as step localization (finding where each step occurs in the video), action segmentation (parsing the video into segments corresponding to steps). Scene context: M (task-/domain-specific) + interaction context: M (step-wise human–object actions) → tasks: action segmentation, step localization, instruction understanding.

Table 2. Popular benchmarks for action recognition. In the observation columns, we nominate which context contributes most to HAR performance: Low—L, Medium—M, High—H.

Benchmark	Size/Duration	Data	Observations
MSR-VTT [99]	10k YouTube videos; 200k descriptions (20 captions per clip); 40 h	Videos (10–30 s each) + manually labeled captions (avg 10 words/caption)	Standard benchmark for video → text retrieval and video question answering (Video QA). Scene context: H (diverse environments, activities) + interaction context: L (implicit, coarse) → tasks: video–text retrieval, Video QA, captioning.
Youcook2 [98]	2k videos; 176 h (avg 5.26 min/video—each video is divided in 7–16 clips)	Cooking instructional videos + one caption per clip (10–20 words)	Used in instructional video understanding tasks (captioning, retrieval). Scene context: M (kitchen, cooking setup) + interaction context: M (human–object, procedural) → tasks: instructional captioning, retrieval, procedure understanding.
Kinetics-600 [95]	474k YouTube videos (avg 10 s/video); 1317 h	Videos + 600 distinct actions classes (e.g., sports, domestic tasks)	Mainly used for video-level action classification. Scene context: H (sports, indoor/outdoor cues) + interaction context: L (coarse labels) → tasks: action classification, representation, pre-training.
Ego4D [96]	3670 h collected from 923 participants across 74 locations and 9 countries (avg 8 min/video)	Egocentric videos (first-person view) + narrations—each narration is a free-form sentence associated with a single timestamp.	Scene context: H (egocentric, environment-aware) + interaction context: H (who–does–what–why, long-term)→ tasks: long-term action anticipation, temporal grounding, QA, NL retrieval.
Something-Something-V2 (SSv2) [97]	170K videos (220k clips with 4–5 s/clip); 276 h	Clips + labels—each clip is labeled with a template caption such as “Putting [something] onto [something else]”.	Widely used for action recognition tasks—specifically, video-level classification of fine-grained human–object interactions (e.g., “Pushing [something] from left to right”). Scene context: L (background suppressed) + interaction context: H (fine-grained object relations) → tasks: fine-grained, action recognition, interaction-centric classification.
COIN [100]	11k videos; 476 h; 46,354 annotated segments	Instructional videos, covering 180 distinct tasks across 12 different domains (e.g., vehicles, gadgets, household items) + annotations	It supports tasks such as step localization (finding where each step occurs in the video), action segmentation (parsing the video into segments corresponding to steps). Scene context: M (task-/domain-specific) + interaction context: M (step-wise human–object actions) → tasks: action segmentation, step localization, instruction understanding.

The first category of solutions is based on early fusion, referred to as the “shared-type” architecture by Luo et al. [101], where video and language modalities are fused together at the initial stages of encoding. This approach was used in one of the earliest instances of VLMs applied to HAR, namely, VideoBERT [102], which learns unified video–language representations through self-supervised pre-training on large-scale videos from YouTube paired with their captions using a masked token completion strategy. A key limitation of VideoBERT is its tendency to lose fine-grained, frame-level visual and temporal details, such as subtle object movements or minor scene changes.

ActBERT [103] addresses this limitation by explicitly modeling global actions and local regional objects alongside linguistic descriptions. Its training objectives include masked language modeling, masked action classification, masked object classification, and cross-modal matching to better align video and text representations. Pre-trained on large instructional datasets like HowTo100M [104] (a data sample—caption + visual—can be seen in Figure 5), ActBERT was evaluated on challenging tasks such as action step localization (CrossTask [105]) and action segmentation (COIN [100]), demonstrating the efficacy of incorporating detailed local cues.

The second category follows a late-fusion paradigm, employing independent video and text encoders with no cross-attention during unimodal encoding. UniVL [101] exemplifies this approach with separate unimodal encoders, a cross-modal encoder, and a decoder trained using coarse-grained contrastive, fine-grained masked modeling, and generative objectives. It was evaluated on YouCook2 [98] for tasks such as video–text retrieval and caption generation.

VideoCLIP [106] extends this paradigm using a dual-encoder structure trained solely with an InfoNCE contrastive objective and derives global embeddings via average token pooling rather than a [CLS] token, preserving token-level features useful for temporal localization. Frozen in Time [107] maintains independent encoders but unifies image and video pre-training by treating images as single-frame videos and progressively increasing temporal context through a curriculum strategy. Despite these advances, coarse global alignment remains a limitation of dual-encoder methods, motivating the development of X-CLIP [108], which introduces both coarse (video–sentence) and fine-grained (frame–word) alignment, cross-granular contrasts, and an Attention over Similarity Matrix mechanism. More recently, the Perception Encoder family [109] has demonstrated that scaled vision–language contrastive pre-training combined with intermediate-layer language and spatial alignment can achieve strong performance with a relatively simple design.

The next paradigm, unified multimodal transfer, or fusion-in-the-backbone architectures, integrates visual and linguistic modalities directly within the model, allowing cross-modal interactions to occur during encoding rather than only at the output level. Here, Flamingo [43] takes a generative LLM-centric approach, connecting frozen visual and language backbones (e.g., Chinchilla 70B [110]) via gated cross-attention layers, where the GATE mechanism blocks the influence of the visual input on the LLM and gradually increases it during training, preventing the model from becoming destabilized. EgoVLPv2 [111], designed for egocentric video understanding, can flexibly switch between dual-encoder and fully fused modes via a gating scalar, enabling both efficient encoding and detailed action reasoning. Extending this approach to long-term action prediction, PALM [112] decomposes tasks into action recognition, captioning, and LLM-based action anticipation modules, leveraging in-context learning to predict future actions in Ego4D [96] and EPIC-KITCHENS [113] datasets. At larger scales, InternVL [114] aligns a six-billion-parameter Vision Transformer (InternViT) with a substantial LLM (QLLaMA, ≈8B) using cross-attention layers to produce fully LLM-compatible embeddings, achieving state-of-the-art performance on Kinetics-600 [95] and Something-Something V2 [97]. Tarsier2-7B builds upon InternVL, incorporating denser latent queries and more sophisticated temporal modeling to enhance subtle action dynamics, supporting both contrastive (with cross-modal fused representations) and generative downstream tasks.

The final category extends pre-trained LLMs with visual inputs, leveraging frozen linguistic priors while integrating vision through lightweight adapters or projection modules. mPLUG-2 [115] adopts a modular design that disentangles spatial and temporal features via a dual-vision encoder, using shared layers and cross-attention to align visual and textual representations in a common semantic space. BLIP-2 [50] follows a model-level bootstrapping strategy, connecting frozen vision encoders to frozen LLMs through a lightweight Q-Former that produces compact query embeddings interpretable by the LLM. This design enables competitive action recognition and multimodal generation while requiring up to ≈$54\times$ fewer trainable parameters than Flamingo variants, highlighting the efficiency of leveraging pre-trained linguistic knowledge for structured reasoning over complex visual dynamics.

Table 3. Selected human action recognition solutions that benefited from VLM captioning. R@n = recall at n; FT = fine-tuned; ZS = zero shot; Acc = accuracy.

Solution	VLM (Visual Language Model/Architecture)	Benchmark and Performance (Metric and Value)	Improvement/Key Contribution
Luo et al., 2021 [101]	UniVL (Text Encoder + Video Encoder); Cross-Modal Encoder; Decoder	Video → Text Retrieval on YouCook2 (FT): R@1: 28.9%; R@10: 70.0%. Video Captioning on YouCook2 (FT): BLEU-4: 17.35, CIDEr: 1.81, METEOR: 22.35.	Early cross-modal VLM; objectives: retrieval, captioning; fine-grained semantics limited by caption quality (e.g., noisy ASR in HowTo100M); no LLM → limited contextual reasoning.
Xu et al., 2021 [106]	VideoCLIP (CLIP + Video Encoder)	Text → Video Retrieval on YouCook2 (ZS): R@1: 22.7, R@5: 50.4, R@10: 63.1. Action Segmentation on COIN (ZS): Accuracy 58.9%.	Contrastive video–text model; objectives: retrieval, zero-shot transfer; fine-grained reasoning constrained by pre-training captions; limited temporal modeling.
Alayrac et al., 2022 [43]	Flamingo (Frozen Vision Encoder + LLM)	VQA on MSRVTT (ZS): Top-1: 19.2%; (32 shots): Top-1: 31.0%.	LLM-based VLM enabling in-context few-shot learning; objectives: retrieval, captioning, QA; distinguishes similar actions via compositional/contextual reasoning; prompt-sensitive, high compute, weaker for strict classification.
Pramanick et al., 2023 [111]	EgoVLPv2 (TimeSformer + RoBERTa-B + Cross-Attention Gated Fusion)	Video → Text Retrieval on EgoMCQ (ZS): Inter Acc. 91.0%, Intra Acc. 60.9%. Video QA on EgoTaskQA head-tuned: Mean Acc.: 36.53%.	Egocentric VLM pre-trained on Ego4D; objectives: retrieval, QA, grounding; fine-grained semantics via hand/object and temporal context; relies on dense aligned narrations.
Xu et al., 2023 [115]	mPLUG-2 (Dual-Vision Encoder + Cross-Attention Fusion + Shared Decoder)	Text → Video Retrieval on MSRVTT (ZS): R@1: 47.1; (FT): R@1: 53.1. Video Captioning MSRVTT (FT): CIDEr: 80.3; BLEU-4: 57.8. Video QA on MSRVTT-QA (FT): Acc. 48.0.	Video–language model exploiting cross-modal correlations; objectives: retrieval, captioning, QA; fine-grained semantics via object–actor–scene grounding; needs careful alignment in multi-task supervision.
Chen et al., 2024 [114]	InternVL (InternViT-6B + Language Middleware + LLM Decoder)	Text → Video Retrieval on MSRVTT (ZS): R@1: 46.3. Video Classification on Kinetics-600 (ZS): 78.8% (Top-1/5 average).	Large-scale multimodal VLM with LLM alignment; objectives: cross-modal matching, generative captioning; fine-grained semantics limited by lack of temporal/action supervision; weaker for long-horizon reasoning.
Bolya et al., 2025 [109]	Perception Encoder (PE) (Scaled Vision Encoder + CLIP-Like Pre-Training + Video Data Engine)	Video Classification on Kinetics-400 (ZS, 8 frames): Top-1: 76.9%. Kinetics-600 (ZS, 8 frames): Top-1: 76.1%. Text→Video retrieval on MSR-VTT (ZS): 51.2 R@1.	Scaled vision encoder with CLIP-style pre-training; objectives: zero-shot classification, retrieval; limited fine-grained multimodal grounding; coarse linguistic and interaction supervision.
Yuan et al., 2025 [116]	Tarsier2-7B (VLM with 7B Parameters, Initialized from Qwen2-VL)	VQA EgoTaskQA (FT): 77.5% Exact Match.	Large-scale video–text pre-training with temporal alignment; objectives: retrieval, QA, captioning; detailed grounding via object–actor–scene; susceptible to hallucination and subtle visual ambiguities.

Table 3. Selected human action recognition solutions that benefited from VLM captioning. R@n = recall at n; FT = fine-tuned; ZS = zero shot; Acc = accuracy.

Solution	VLM (Visual Language Model/Architecture)	Benchmark and Performance (Metric and Value)	Improvement/Key Contribution
Luo et al., 2021 [101]	UniVL (Text Encoder + Video Encoder); Cross-Modal Encoder; Decoder	Video → Text Retrieval on YouCook2 (FT): R@1: 28.9%; R@10: 70.0%. Video Captioning on YouCook2 (FT): BLEU-4: 17.35, CIDEr: 1.81, METEOR: 22.35.	Early cross-modal VLM; objectives: retrieval, captioning; fine-grained semantics limited by caption quality (e.g., noisy ASR in HowTo100M); no LLM → limited contextual reasoning.
Xu et al., 2021 [106]	VideoCLIP (CLIP + Video Encoder)	Text → Video Retrieval on YouCook2 (ZS): R@1: 22.7, R@5: 50.4, R@10: 63.1. Action Segmentation on COIN (ZS): Accuracy 58.9%.	Contrastive video–text model; objectives: retrieval, zero-shot transfer; fine-grained reasoning constrained by pre-training captions; limited temporal modeling.
Alayrac et al., 2022 [43]	Flamingo (Frozen Vision Encoder + LLM)	VQA on MSRVTT (ZS): Top-1: 19.2%; (32 shots): Top-1: 31.0%.	LLM-based VLM enabling in-context few-shot learning; objectives: retrieval, captioning, QA; distinguishes similar actions via compositional/contextual reasoning; prompt-sensitive, high compute, weaker for strict classification.
Pramanick et al., 2023 [111]	EgoVLPv2 (TimeSformer + RoBERTa-B + Cross-Attention Gated Fusion)	Video → Text Retrieval on EgoMCQ (ZS): Inter Acc. 91.0%, Intra Acc. 60.9%. Video QA on EgoTaskQA head-tuned: Mean Acc.: 36.53%.	Egocentric VLM pre-trained on Ego4D; objectives: retrieval, QA, grounding; fine-grained semantics via hand/object and temporal context; relies on dense aligned narrations.
Xu et al., 2023 [115]	mPLUG-2 (Dual-Vision Encoder + Cross-Attention Fusion + Shared Decoder)	Text → Video Retrieval on MSRVTT (ZS): R@1: 47.1; (FT): R@1: 53.1. Video Captioning MSRVTT (FT): CIDEr: 80.3; BLEU-4: 57.8. Video QA on MSRVTT-QA (FT): Acc. 48.0.	Video–language model exploiting cross-modal correlations; objectives: retrieval, captioning, QA; fine-grained semantics via object–actor–scene grounding; needs careful alignment in multi-task supervision.
Chen et al., 2024 [114]	InternVL (InternViT-6B + Language Middleware + LLM Decoder)	Text → Video Retrieval on MSRVTT (ZS): R@1: 46.3. Video Classification on Kinetics-600 (ZS): 78.8% (Top-1/5 average).	Large-scale multimodal VLM with LLM alignment; objectives: cross-modal matching, generative captioning; fine-grained semantics limited by lack of temporal/action supervision; weaker for long-horizon reasoning.
Bolya et al., 2025 [109]	Perception Encoder (PE) (Scaled Vision Encoder + CLIP-Like Pre-Training + Video Data Engine)	Video Classification on Kinetics-400 (ZS, 8 frames): Top-1: 76.9%. Kinetics-600 (ZS, 8 frames): Top-1: 76.1%. Text→Video retrieval on MSR-VTT (ZS): 51.2 R@1.	Scaled vision encoder with CLIP-style pre-training; objectives: zero-shot classification, retrieval; limited fine-grained multimodal grounding; coarse linguistic and interaction supervision.
Yuan et al., 2025 [116]	Tarsier2-7B (VLM with 7B Parameters, Initialized from Qwen2-VL)	VQA EgoTaskQA (FT): 77.5% Exact Match.	Large-scale video–text pre-training with temporal alignment; objectives: retrieval, QA, captioning; detailed grounding via object–actor–scene; susceptible to hallucination and subtle visual ambiguities.

A fundamental distinction exists between video-only and multimodal models in action recognition. While both can be evaluated on the same benchmarks, multimodal models are typically assessed in zero-shot settings. For example, on Kinetics-400, VideoMAE [117] achieves 87.4% Top-1 accuracy when fine-tuned, whereas the Perception Encoder (PE) reaches approximately 76.9% Top-1 accuracy in zero-shot evaluation. This underscores that direct accuracy comparisons with fully fine-tuned video-only models are not strictly valid; multimodal pre-training emphasizes generalization to unseen categories rather than peak performance on a single task. Consequently, unimodal–multimodal comparisons are meaningful only for action classification with identical labels and metrics; for tasks such as language-driven retrieval, video captioning, video question answering, or instructional step localization, unimodal baselines are inapplicable.

This distinction persists under fine-tuning. In video-only models, adaptation to a new dataset primarily addresses domain shift; pre-trained representations (e.g., VideoMAE) are fine-tuned on smaller datasets such as UCF101 [118] to match the target distribution while preserving the same task. In multimodal models, fine-tuning additionally entails a shift in learning objectives. For instance, UniVL transitions from reconstructing raw text during pre-training to generating human-annotated captions on YouCook2, while mPLUG specializes its modular components for task-specific video understanding objectives. More broadly, multimodal fine-tuning goes beyond domain adaptation by progressively aligning semantically distinct modalities, constituting a deeper transformation of the model’s representational space.

7. VLMs in Violence Detection

In this section, we review recent advancements in video anomaly detection (VAD) and video violence detection (VVD), emphasizing methods that harness the multimodal reasoning and contextual understanding capabilities of large language models (LLMs) and vision–language models (VLMs). These approaches are categorized by their contributions to training efficiency, structural design, and cross-modal reasoning.

First, we note that specific benchmarks have been developed. In Table 4, a brief description of the violence/anomaly detection benchmarks is presented. Examples of images from those benchmarks can be seen in Figure 6.

A major innovation trend involves adapting frozen VLMs for VAD without costly parameter tuning, relying instead on prompt engineering or learned guiding questions [119,120,121]. ASK-HINT [120] refines this paradigm by introducing fine-grained, action-centric prompts that align linguistic reasoning with discriminative visual cues. Through Semantic Compression, it synthesizes a compact set of representative questions that enhance interpretability and reduce hallucination effects. VERA (Verbalized Learning) [121] further extends training-free adaptation by transforming the reasoning process into a learnable dialogue between two VLMs, where optimized guiding questions are iteratively refined. During inference, these questions guide segment-level scoring and are refined via temporal and scene context fusion, allowing explainable and fine-grained anomaly detection.

Other works emphasize domain adaptation, bias reduction, and efficient explanation generation. Holmes-VAD [122] introduces the VAD-Instruct50k benchmark—an instruction-tuning dataset with semi-automated annotations—to train multimodal LLMs for explainable VAD. A Temporal Sampler accelerates inference from 32.82 s to 4.24 s per video, while Projector + LoRA fine-tuning ensures lightweight adaptability. VIVID [123] addresses interpretability and fairness in VVD by leveraging an external knowledge base of 48 violence definitions (from sources such as Cambridge and Oxford) and employing multimodal retrieval and optical flow-based key frame selection. It also reorders visual queries to mitigate the “lost-in-the-middle” effect common in long visual sequences. PiercingEye [124] advances geometric representation learning through Dual-Space Learning, combining Euclidean and Hyperbolic GCNs to model spatial and hierarchical event contexts. Its Ambiguous Event Text Generation module produces pseudo-supervisory textual descriptions of uncertain scenes, while Hyperbolic Vision–Language Guided Loss prioritizes challenging negatives using text similarity-based weighting.

Further contributions focus on multimodal integration and generalization. ViCap-AD [125] introduces a weakly supervised framework that fuses video captions generated by CLIP4Clip with BERT-based textual features under a Multiple-Instance Learning scheme. The final anomaly score combines Classification Score (CS) and Multiple-Instance Learning Ranking Loss for improved stability. The Multimodal Dual-Stream VAD [17] model fuses video, audio, and text modalities via Abnormal-Aware Context Prompts. These encode prior knowledge and anomalous cues into textual representations, enabling fine-grained discrimination, while a coarse stream employs contrastive optimization for global audio–visual alignment. The language-based VAD (LAVAD) [119] framework introduces a training-free approach to video anomaly detection by combining frozen LLMs and VLMs. It first generates frame-level captions using an off-the-shelf model, then refines them through Image–Text Caption Cleaning to reduce noise. Finally, an LLM-based Anomaly Scoring module aggregates the cleaned captions into temporal summaries and prompts the LLM to produce anomaly scores, enabling effective detection without model fine-tuning.

Table 4. Summary of common datasets used for video anomaly and violence detection.

Dataset	Clips	Anomaly/Violence Types	Source	Key Characteristics
UCF-Crime [126]	1900	13 anomalies (abuse, fighting, shooting, etc.)	Surveillance (indoor/outdoor)	Long, untrimmed videos; video-level training, frame-level testing.
XD-Violence [127]	4754	6 classes (abuse, accidents, riots, etc.)	Movies, YouTube, CCTV	Large-scale; includes audio; weakly labeled untrimmed videos.
RWF-2000 [128]	1600 train 400 test	Violent vs. non-violent	YouTube surveillance	Uniform 5 s clips at 30 fps.
Movies [129]	200	Fights vs. non-fight scenes	Action films, sports	Diverse scenes; manually labeled.
Surv. Fight [130]	300	Fights (kicks, fists, wrestling) vs. non-fight	YouTube (cafes, streets, buses)	Short clips (2 s) from real-world surveillance.
Hockey [129]	1000	Fights vs. non-fights	NHL games	Dynamic setting; violent and normal actions look similar.

Table 4. Summary of common datasets used for video anomaly and violence detection.

Dataset	Clips	Anomaly/Violence Types	Source	Key Characteristics
UCF-Crime [126]	1900	13 anomalies (abuse, fighting, shooting, etc.)	Surveillance (indoor/outdoor)	Long, untrimmed videos; video-level training, frame-level testing.
XD-Violence [127]	4754	6 classes (abuse, accidents, riots, etc.)	Movies, YouTube, CCTV	Large-scale; includes audio; weakly labeled untrimmed videos.
RWF-2000 [128]	1600 train 400 test	Violent vs. non-violent	YouTube surveillance	Uniform 5 s clips at 30 fps.
Movies [129]	200	Fights vs. non-fight scenes	Action films, sports	Diverse scenes; manually labeled.
Surv. Fight [130]	300	Fights (kicks, fists, wrestling) vs. non-fight	YouTube (cafes, streets, buses)	Short clips (2 s) from real-world surveillance.
Hockey [129]	1000	Fights vs. non-fights	NHL games	Dynamic setting; violent and normal actions look similar.

Results Comments

Table 5 shows that the integration of textual information, whether through detailed captions, formalized rules, or targeted prompts, consistently demonstrates a strong positive impact on the performance and interpretability of video anomaly detection (VAD) models, often surpassing visual-only baselines or abstract language models.

ASK-HINT [120] leverages fine-grained, action-centric prompts to better align linguistic reasoning with visual cues, achieving state-of-the-art AUC results through structured prompting. Holmes-VAD [122] employs instruction tuning with paired visual–textual explanations, markedly improving interpretability and human-aligned reasoning metrics. VERA [121] improves reasoning by refining prompts into detailed, anomaly-focused questions, enabling fine-grained frame-level scoring with temporal and contextual awareness.

The comparison in Table 6 highlights the strong performance achieved by a diverse set of methods. The Holmes-VAD [122] model, which is instruction-tuned on a purpose-built dataset for explainable VAD, achieves state-of-the-art results on both benchmarks. In the case of multimodal VAD [17], the use of Abnormal-Aware Context Prompts embeds semantic and anomaly-related knowledge into the text, guiding detection with meaningful context. This multimodal synergy enhances sensitivity to subtle and complex anomalies, achieving superior fine-grained accuracy. At the same time, the strong performance of diverse methods such as the dual-space geometric PiercingEye [124] and the verbalized-learning VERA [121] indicates that, while large-scale instruction tuning is currently SOTA, specialized architectural innovations that require less data and computation offer highly competitive and potentially more practical alternatives.

Table 5. Impact of captioning and textual guidance on VLM/LLM-based VAD performance. Incorporating textual information—via detailed captions, formalized rules, or targeted prompts—consistently enhances both the performance and interpretability of video anomaly detection models, often outperforming visual-only approaches or standalone language models. AUC = Area under Curve; AP = Average Precision; JA = Judgement Accuracy; CP = Content Perception; AE = Anomaly Explanatory.

Method	Datasets	Metrics (Values)	Effect of Captioning/Text Guidance
Holmes-VAD [122]	UCF-Crime, XD-Violence	AUC: 89.51% (UCF); AP: 90.67% (XD); JA: 86.0%; CP: 61.2%; AE: 51.9%	Instruction tuning (VAD-Instruct50k) greatly boosted interpretability: JA (86.0 vs. 65.1), CP (61.2 vs. 11.6), AE (51.9 vs. 15.9).
ASK-HINT [120]	UCF-Crime, XD-Violence	AUC (UCF): 89.83%; AUC (XD): 90.31%	Fine-grained prompting (6 prompts) outperformed full-prompt baseline (67.17%), improving AUC by 22.6%.
VERA  [121]	UCF-Crime, XD-Violence	AUC (UCF): 86.55%; AUC (XD): 88.26%	Guiding question prompts raised AUC from 78.81% to 86.55%, highlighting textual reasoning benefits.
VIVID [123]	Movies, Surv. Fight, RWF-2000, Hockey, XD-Violence	Acc: Movies 0.985; Surv.fight: 0.736; RWF-2000: 0.797; Hockey: 0.954; 0.985; XD 0.826	Violence definition database reduced bias and improved accuracy (up to +0.32).
ViCap-AD [125]	UCF-Crime, XD-Violence	AUC (UCF): 87.20%; AUC (XD): 85.02%	CLIP4Clip captions + Multiple-Instance Learning losses increased AUC by 1.72% (87.20 vs. 85.48) on UCF crime.
PiercingEye [124]	XD-Violence, UCF-Crime	AP: 88.82% (XD); AUC: 86.64% (UCF)	Hyperbolic Vision-Language Guided Loss with Ambiguous Event Text Generation text raised AP by 1.21% (87.61 → 88.82).
Multimodal VAD  [17]	UCF-Crime, XD-Violence	AUC: 87.96% (UCF); AP: 86.32% (XD)	Adding text in V+A+T setup improved AP by at least 4% over video-only models for XD Violence.
LAVAD [119]	UCF-Crime, XD-Violence	AUC: 80.28% (UCF); AUC: 85.36% (XD)	Removing LLM-based anomaly scoring dropped AUC by −7.58% (80.28 → 72.70) on UCF-Crime.

Table 5. Impact of captioning and textual guidance on VLM/LLM-based VAD performance. Incorporating textual information—via detailed captions, formalized rules, or targeted prompts—consistently enhances both the performance and interpretability of video anomaly detection models, often outperforming visual-only approaches or standalone language models. AUC = Area under Curve; AP = Average Precision; JA = Judgement Accuracy; CP = Content Perception; AE = Anomaly Explanatory.

Method	Datasets	Metrics (Values)	Effect of Captioning/Text Guidance
Holmes-VAD [122]	UCF-Crime, XD-Violence	AUC: 89.51% (UCF); AP: 90.67% (XD); JA: 86.0%; CP: 61.2%; AE: 51.9%	Instruction tuning (VAD-Instruct50k) greatly boosted interpretability: JA (86.0 vs. 65.1), CP (61.2 vs. 11.6), AE (51.9 vs. 15.9).
ASK-HINT [120]	UCF-Crime, XD-Violence	AUC (UCF): 89.83%; AUC (XD): 90.31%	Fine-grained prompting (6 prompts) outperformed full-prompt baseline (67.17%), improving AUC by 22.6%.
VERA  [121]	UCF-Crime, XD-Violence	AUC (UCF): 86.55%; AUC (XD): 88.26%	Guiding question prompts raised AUC from 78.81% to 86.55%, highlighting textual reasoning benefits.
VIVID [123]	Movies, Surv. Fight, RWF-2000, Hockey, XD-Violence	Acc: Movies 0.985; Surv.fight: 0.736; RWF-2000: 0.797; Hockey: 0.954; 0.985; XD 0.826	Violence definition database reduced bias and improved accuracy (up to +0.32).
ViCap-AD [125]	UCF-Crime, XD-Violence	AUC (UCF): 87.20%; AUC (XD): 85.02%	CLIP4Clip captions + Multiple-Instance Learning losses increased AUC by 1.72% (87.20 vs. 85.48) on UCF crime.
PiercingEye [124]	XD-Violence, UCF-Crime	AP: 88.82% (XD); AUC: 86.64% (UCF)	Hyperbolic Vision-Language Guided Loss with Ambiguous Event Text Generation text raised AP by 1.21% (87.61 → 88.82).
Multimodal VAD  [17]	UCF-Crime, XD-Violence	AUC: 87.96% (UCF); AP: 86.32% (XD)	Adding text in V+A+T setup improved AP by at least 4% over video-only models for XD Violence.
LAVAD [119]	UCF-Crime, XD-Violence	AUC: 80.28% (UCF); AUC: 85.36% (XD)	Removing LLM-based anomaly scoring dropped AUC by −7.58% (80.28 → 72.70) on UCF-Crime.

Table 6. Performance of advanced VAD methods on key datasets. AUC = Area under Curve; AP = Average Precision.

Method	Primary Approach	UCF-Crime (AUC %)	XD-Violence (AP %)
LAVAD  [119]	Training-Free (VLM Captions + LLM)	80.28	62.01
VERA  [121]	Verbalized Learning (Frozen VLM)	86.55	70.54
PiercingEye [124]	Dual-Space Geometric Learning	86.64	88.82
Multimodal-AD [17]	Dual-Stream Network (Coarse: V+A, Fine: T)	87.96	86.32
Holmes-VAD [122]	Instruction-Tuned (Explainable VAD)	89.51	90.67

Table 6. Performance of advanced VAD methods on key datasets. AUC = Area under Curve; AP = Average Precision.

Method	Primary Approach	UCF-Crime (AUC %)	XD-Violence (AP %)
LAVAD  [119]	Training-Free (VLM Captions + LLM)	80.28	62.01
VERA  [121]	Verbalized Learning (Frozen VLM)	86.55	70.54
PiercingEye [124]	Dual-Space Geometric Learning	86.64	88.82
Multimodal-AD [17]	Dual-Stream Network (Coarse: V+A, Fine: T)	87.96	86.32
Holmes-VAD [122]	Instruction-Tuned (Explainable VAD)	89.51	90.67

8. VLMs in Contextual Emotions Recognition

Understanding human states such as emotions, intentions, and actions is a central challenge in computer vision. These states are critical not only for applications in human–computer interaction, assistive technologies, and surveillance, but also for advancing fundamental research on how machines perceive and reason about people. Emotions play a vital role in communication and social dynamics, and recognizing them from visual data has therefore become an active research field [131,132,133].

Traditional approaches in visual recognition have primarily focused on localized cues, most often relying on cropped facial images or isolated body poses. This face-centered perspective has been strengthened by the rise of large models.

While effective in constrained settings, the methods focused on a specific part (like the ones using only the face expression) sometimes fall short of capturing how humans naturally perceive others: by integrating multiple signals including body posture, gestures, social interactions, and background context. Context-aware emotion recognition (CAER) [134] and datasets such as EMOTIC have highlighted that environmental and scene information strongly influences the interpretation of human states. For example, the same facial expression may be perceived differently when situated in a playground versus a conflict zone. This contextualization is therefore essential for achieving reliable recognition.

To visually illustrate various problems in emotion inference from visual data, in Figure 7, we present images from popular databases. The images are clustered in rows according to theme. Details on the datasets may be found in

Table 7. Databases used in contextual emotion recognition that benefited from VLM captioning. The top part lists databases focuses on body language and the face, while the bottom sets are focused on face expressions.

Model	Size	Data	Observations
BOLD [135]	9876 video clips and 13,239 annotated human characters; 70/10/20 split.	26 emotion classes.	In-the-wild body language dataset focusing on motion and keypoints. Designed for bodily expression recognition.
HECO [136]	9385 images with 19,781 annotated agents (multiple people per image).	8 discrete emotions + VAD dimensional scores.	Context-rich static image dataset focusing on multiple agents and their interactions; annotated by experts. Contains occlusions and ambiguous cases for realism.
EMOTIC [137]	23,571 images and 34,320 annotated people; 70/10/20 split.	26 emotion classes.	In-the-wild dataset emphasizing contextual cues beyond facial expressions. Highly imbalanced, reflecting real-world emotion distributions.
CAER/CAER-S [138]	13,201 video clips (∼1,107,877 frames) and ∼70 k static images (CAER-S).	7 emotion classes.	Collected from TV shows, providing rich contextual emotion cues from human interactions in diverse scenarios.
EMOSET [139]	3.3 M weakly labeled images; 118 K human-annotated subset.	8 emotion categories based on Mikel’s Model.	Large-scale visual emotion dataset from social media and artistic sources, emphasizing contextual and affective content.
DFEW [131]	16,372 facial expression clips from 1500 movies.	7 emotion classes.	Dynamic facial expression recognition dataset in the wild; focuses exclusively on facial cues from diverse cinematic sources.
FERV39K [140]	38,935 video clips (>1 M frames) from 4K raw videos across 22 scene types.	7 emotion classes.	Large-scale in-the-wild dataset emphasizing diverse multi-scene contexts and dynamic facial cues.
AFEW [141]	1426 video clips from movies featuring 330 subjects aged 1–70.	7 emotion classes.	One of the earliest in-the-wild benchmarks for emotion recognition. Acted, expressions include real-world variability in lighting, occlusion, and background.
RAF-DB [142]	29,672 images.	7 basic + 12 compound emotions.	In-the-wild Internet images; crowd-annotated with quality control.
AffectNet [143]	450 k images (400 k manually annotated).	8 categorical; valence/arousal (dimensional).	Large-scale, Internet-sourced; dual annotation scheme (categorical + dimensional).
ExpW [144]	91,793 faces (manually annotated).	7 categorical emotions.	Large-scale, Internet-sourced.

and Table 8.

When compared to the classical solution that is trained within a single database, more recent versions have benefited from the development of large models. Recent advances in VLMs offer a powerful framework for addressing various challenges in this domain too, challenges that mostly stem from data domain limitations. VLMs such as CLIP [9], BLIP [50], and LLaVA [46] learn joint representations of images and text from massive collections of paired data. By aligning vision and language in a shared space, they enable models to capture not only local objects or body features, but also broader semantic cues present in the surrounding environment. Solutions for contextual emotion recognition that benefited from VLM captioning may be found in Table 9.

Regarding contextual emotion recognition (Table 9), a point of analysis is the trade-offs between interpretability, performance, and computational cost across different model categories. LLaVA-based approaches [16,145] and Collaborative VLM-LLM systems [146,147] demonstrate strong contextual understanding and reasoning capabilities. By employing two-stage processing, caption generation followed by classification, or iterative refinement between modalities, these models provide interpretable narrative captions and nuanced emotion predictions. However, this depth comes at the cost of increased latency and computational complexity, with the risk of caption generation errors propagating to the final classification. Similarly, large-scale LVLMs [148] leverage in-context learning and Chain-of-Thought prompting to achieve high-quality reasoning with few-shot learning, but they require significant computational resources (e.g., 8B parameters) and depend heavily on high-quality prompt generation.

In contrast, off-the-shelf evaluation approaches [149] offer rapid deployment and strong zero-shot capabilities without fine-tuning, though they often yield lower absolute performance compared to specialized models and raise privacy or cost concerns due to API dependencies. Instruction-tuned models (e.g., [150]) strike a balance by aligning visual features with emotional concepts through diverse training signals, yet their success is contingent on the computationally expensive generation of high-quality instruction datasets. Overall, while VLM integration consistently yields substantial performance gains (e.g., 5–15% F1 improvement) on typical benchmarks, researchers must weigh the benefits of explicit reasoning and interpretability against the higher resource demands of these complex architectures.

VLMs in Face Expression Recognition

Inference of emotion from face expression recognition is the most well-established direction. A thorough review may be found in the work of Kalateh et al. [29]. Examples of illustrative images are found in the top row of Figure 7. In this case, the focus is on the portrait, as often the background is too loosely correlated to the emotion of the character.

A listing of the methods is given in Table 10. In summary, recent advancements in facial expression recognition (FER) have demonstrated a clear shift toward leveraging VLMs and Large Multimodal Models, which have improved contextual understanding, reasoning, and generalization across complex affective scenarios.

The integration of VLMs and LLMs has redefined facial expression recognition by enabling models to move beyond isolated facial features toward context-aware, semantically grounded, and interpretable emotional reasoning. These approaches demonstrate key advantages. Methods like EmoGist [151] and SoVTP [152] demonstrate that emotional interpretation can successfully rely on global scene reasoning rather than facial features alone, emphasizing the role of context in emotion recognition. Techniques such as Exp-CLIP [153] and FEALLM [154] connect the gap between visual features and language-driven emotional concepts by leveraging semantic alignment and knowledge transfer. ExpLLM [155] and GPT-based micro-expression systems [156] highlight how Chain-of-Thought reasoning enables interpretability and trust in FER systems. By using flexible, descriptive, and high-dimensional representations of emotions, EmoCapCLIP [157] shows the value of using natural language captions to capture the complexity of human affect beyond categorical labels. The concept has been expanded further than classical emotion recognition to depression detection [158], with contextualization and fine-grained emotional annotation being applied to depression video data.

In summary, for FER solutions (Table 10), the landscape is dominated by approaches that leverage pre-trained vision–language knowledge. CLIP-based temporal models [159] and fine-grained captioning models [160] excel at capturing dynamic facial movements and subtle expression changes, significantly outperforming baselines on video benchmarks. However, temporal modeling adds computational complexity, and generic CLIP features may sometimes miss micro-movements. Multi-prompt learning approaches [15] and task-aware alignment models [153] offer efficient alternatives by learning rich emotion representations or aligning features with emotion-specific semantic spaces (often using action units) without explicit caption generation, achieving very strong results on static datasets like RAF-DB.

Hybrid expert systems [161] and rule-enhanced LLM systems [156] push performance further by specializing in different emotion aspects or amplifying weak signals (like micro-expressions) through expert rules. While these systems achieve strong accuracy on specific benchmarks, they introduce significant complexity into the model architecture and hyperparameter tuning. Example-based cluster analysis [151] offers a computationally efficient inference alternative by pre-generating context descriptions, though it risks missing rare expressions that have not been seen by the VLM. Ultimately, the choice of model involves balancing the need for temporal generalization and interpretability (favored by captioning and AU-based models) against model complexity and computational constraints.

Table 8. Multimodal emotion databases.

Model	Size	Data	Observations
MELD [162]	13,000 utterances across 1433 dialogues from 1000+ speakers.	7 categ. emotions + sentiment polarity (pos.–neut.–neg.).	Multimodal dataset (audio, visual, text) derived from the TV series Friends, designed for multi-party conversational emotion recognition at the utterance level.
VCR [163]	110,000 movie scenes forming ∼290,000 question–answer pairs.	4 multiple-choice answers + 4 rationales per question.	Multimodal (image + text) dataset targeting visual common-sense reasoning, requiring understanding of context, intentions, and affect.
MER2023 [164]	∼3373 labeled samples and >73,000 unlabeled samples for semi-supervised learning.	6 discrete emotions (neutral, anger, happiness, sadness, worry, surprise).	Multimodal dataset (video, audio, text) designed for robust emotion recognition under noisy and semi-supervised conditions. Includes multiple tracks: MER-MULTI, MER-NOISE, MER-SEMI.
MAFW [165]	10,045 video–audio clips with textual affective captions.	11 basic and 32 compound emotions.	Multimodal dataset combining video, audio, and text for emotion recognition in complex natural contexts. Supports hierarchical affective labeling.

Table 8. Multimodal emotion databases.

Model	Size	Data	Observations
MELD [162]	13,000 utterances across 1433 dialogues from 1000+ speakers.	7 categ. emotions + sentiment polarity (pos.–neut.–neg.).	Multimodal dataset (audio, visual, text) derived from the TV series Friends, designed for multi-party conversational emotion recognition at the utterance level.
VCR [163]	110,000 movie scenes forming ∼290,000 question–answer pairs.	4 multiple-choice answers + 4 rationales per question.	Multimodal (image + text) dataset targeting visual common-sense reasoning, requiring understanding of context, intentions, and affect.
MER2023 [164]	∼3373 labeled samples and >73,000 unlabeled samples for semi-supervised learning.	6 discrete emotions (neutral, anger, happiness, sadness, worry, surprise).	Multimodal dataset (video, audio, text) designed for robust emotion recognition under noisy and semi-supervised conditions. Includes multiple tracks: MER-MULTI, MER-NOISE, MER-SEMI.
MAFW [165]	10,045 video–audio clips with textual affective captions.	11 basic and 32 compound emotions.	Multimodal dataset combining video, audio, and text for emotion recognition in complex natural contexts. Supports hierarchical affective labeling.

Table 9. Solutions for contextual emotion recognition that benefited from VLM captioning. In certain cases, the results are the consequence of specific conditions which are detailed in the cited references. mAP = Mean Average Precision; Acc = accuracy; Rec = recall; Prec = precision.

Solution	VLM	Benchmark and Performance	Improvement
Xenos et al. (2024) [16]	LLaVA (two-stage; caption + classify)	$mA{P}_{EMOTIC}$ = 38.52% $mA{P}_{BoLD}$ = 26.66% $Ac{c}_{CAE{R}_S}$ = 93.08%	Fuse visual features + the generated text via a Transformer architecture (vision encoder → learnable queries → Q-Former cross + self attention, then classifier)
Etesam et al. (2024) [145]	LLaVA, GPT-4V	$Pre{c}_{EMOTIC}$ = 54.27% $Re{c}_{EMOTIC}$ = 78.40% $F{1}_{EMOTIC}$ = 36.83%	Introduces narrative captioning (NarraCap): generates descriptive captions (who, action, social/physical signals, environment) then feeds to LLM for emotion inference
Lei et al. (2024) [148]	VILA-8B (LVLM few-shot + CoT)	$Pre{c}_{EMOTIC}$ = 57.8% $Re{c}_{EMOTIC}$ = 46.9% $F{1}_{EMOTIC}$ = 29.8% $Pre{c}_{HECO}$ = 48.3% $Re{c}_{HECO}$ = 33.6% $F{1}_{HECO}$ = 35.8%	Uses in-context learning (ICL) and Chain-of-Thought (CoT) prompts generated by GPT-4V for contextual reasoning
Bhattacharyya & Wang (2025) [149]	GPT-4o, CLIP variants	$F{1}_{Emotion6}$ = 63.5% $F{1}_{Abstract}$ = 27.0% $F{1}_{ArtPhoto}$ = 45.0% $F{1}_{Hard}$ = 44.9%	Evaluates off-the-shelf VLMs using reasoning-based prompts for improved emotion prediction accuracy
Xie et al. (2024) [150]	EmoVIT (InstructBLIP + visual instruction tuning)	$Ac{c}_{Emotion6}$ = 57.6% $Ac{c}_{Abstract}$ = 32.3% $Ac{c}_{ArtPhoto}$ = 44.9% $Ac{c}_{WebEmo}$ = 21.1%	Uses GPT-4-generated visual instruction data including captions, categorical, and reasoning forms to fine-tune emotion understanding via instruction following
Zhou et al. (2024) [146]	ViCor (LLM + VLM collaboration)	$Ac{c}_{VCR}$ = 59.8% $Ac{c}_{A-OKVQA}$ = 70.9%	Bridges vision understanding and common-sense reasoning: captions + visual clues are iteratively exchanged between the VLM and LLM to refine emotion and common-sense inference
Lian et al. (2024) [147]	AffectGPT (Multimodal LLM + MER-Caption dataset)	$Ac{c}_{EMER-Fine}$ = 64.56%	Introduces descriptive emotion captions (MER-Caption), enabling free-form emotion reasoning across modalities (vision, audio, text)

Table 9. Solutions for contextual emotion recognition that benefited from VLM captioning. In certain cases, the results are the consequence of specific conditions which are detailed in the cited references. mAP = Mean Average Precision; Acc = accuracy; Rec = recall; Prec = precision.

Solution	VLM	Benchmark and Performance	Improvement
Xenos et al. (2024) [16]	LLaVA (two-stage; caption + classify)	$mA{P}_{EMOTIC}$ = 38.52% $mA{P}_{BoLD}$ = 26.66% $Ac{c}_{CAE{R}_S}$ = 93.08%	Fuse visual features + the generated text via a Transformer architecture (vision encoder → learnable queries → Q-Former cross + self attention, then classifier)
Etesam et al. (2024) [145]	LLaVA, GPT-4V	$Pre{c}_{EMOTIC}$ = 54.27% $Re{c}_{EMOTIC}$ = 78.40% $F{1}_{EMOTIC}$ = 36.83%	Introduces narrative captioning (NarraCap): generates descriptive captions (who, action, social/physical signals, environment) then feeds to LLM for emotion inference
Lei et al. (2024) [148]	VILA-8B (LVLM few-shot + CoT)	$Pre{c}_{EMOTIC}$ = 57.8% $Re{c}_{EMOTIC}$ = 46.9% $F{1}_{EMOTIC}$ = 29.8% $Pre{c}_{HECO}$ = 48.3% $Re{c}_{HECO}$ = 33.6% $F{1}_{HECO}$ = 35.8%	Uses in-context learning (ICL) and Chain-of-Thought (CoT) prompts generated by GPT-4V for contextual reasoning
Bhattacharyya & Wang (2025) [149]	GPT-4o, CLIP variants	$F{1}_{Emotion6}$ = 63.5% $F{1}_{Abstract}$ = 27.0% $F{1}_{ArtPhoto}$ = 45.0% $F{1}_{Hard}$ = 44.9%	Evaluates off-the-shelf VLMs using reasoning-based prompts for improved emotion prediction accuracy
Xie et al. (2024) [150]	EmoVIT (InstructBLIP + visual instruction tuning)	$Ac{c}_{Emotion6}$ = 57.6% $Ac{c}_{Abstract}$ = 32.3% $Ac{c}_{ArtPhoto}$ = 44.9% $Ac{c}_{WebEmo}$ = 21.1%	Uses GPT-4-generated visual instruction data including captions, categorical, and reasoning forms to fine-tune emotion understanding via instruction following
Zhou et al. (2024) [146]	ViCor (LLM + VLM collaboration)	$Ac{c}_{VCR}$ = 59.8% $Ac{c}_{A-OKVQA}$ = 70.9%	Bridges vision understanding and common-sense reasoning: captions + visual clues are iteratively exchanged between the VLM and LLM to refine emotion and common-sense inference
Lian et al. (2024) [147]	AffectGPT (Multimodal LLM + MER-Caption dataset)	$Ac{c}_{EMER-Fine}$ = 64.56%	Introduces descriptive emotion captions (MER-Caption), enabling free-form emotion reasoning across modalities (vision, audio, text)

Table 10. Face expression recognition solutions that benefited from VLM captioning. In certain cases, the results are the consequence of specific conditions which are detailed in the cited references. $UAR$ = Unweighted Average Recall; $WAR$ = Weighted Average Recall.

Solution	VLM	Benchmark and Performance	Improvement
Zhao & Patras (2023) [159]	DFER-CLIP (CLIP + LLM text descriptions)	$UAR/WA{R}_{DFEW}$ = 59.6/71.3%; $UAR/WA{R}_{FERV39k}$ = 41.3/51.6%; $UAR/WA{R}_{MAFW}$ = 38.9/52.6%;	Uses ChatGPT-generated textual behavior descriptions instead of class labels → improved temporal FER generalization
Foteinopoulou & Patras (2024) [166]	EmoCLIP (CLIP + temporal transformer)	$UAR/WA{R}_{DFEW}$ = 58.1/62.1%; $UAR/WA{R}_{FERV39k}$ = 31.3/43.5%; $UAR/WA{R}_{MAFW}$ = 34.3/44.2%; $UAR/WA{R}_{AFEW}$ = 44.3/46.2%;	Uses sample-level captions of expressions and LLM-generated class descriptions for zero-shot video FER
Li et al. (2024) [15]	CLIPER (CLIP + METD)	$UAR/WA{R}_{FERV39k}$ = 41.2/51.4%; $WA{R}_{RAF-DB}$ = 91.8%	Learns multiple text prompts per emotion for richer semantics; better alignment without explicit captions
Chen et al. (2024) [160]	FineCLIPER (CLIP + AdaptERs + Video-LLaVA)	$UAR/WA{R}_{DFEW}$ = 66.0/76.2%; $UAR/WA{R}_{FERV39k}$ = 45.2/54.0%; $UAR/WA{R}_{MAFW}$ = 45.0/56.9%;	Fine-grained dynamic captions from Video-LLaVA improve temporal FER; surpasses CLIP baseline across datasets
Huang et al. (2025) [161]	Emotion-Qwen (Hybrid Expert Mixture)	$UAR/WA{R}_{DFEW}$ = 77.1/62.2%;	Integrates VER dataset for instruction fine-tuning with contextual and causal captions to enhance emotion reasoning
Seoh et al. [151]	Qwen2.5-VL; Aya Vision; InternVL2.5 8B-MPO	$Pre{c}_{FI}$ = 52.9; $Re{c}_{FI}$ = 52.2; $F{1}_{FI}$ = 48.5	EmoGist analyzes clusters of example images to pre-generate multiple context-specific descriptions via the VLM for emotion labels
Zhao et al. [153]	CLIP-L	$UAR/WA{R}_{RAF-DB}$ = 58.7/65.4; $UAR/WA{R}_{AffectNet}$ = 38.4/38.4; $UAR/WA{R}_{DFEW}$ = 40.2/47.1; $UAR/WA{R}_{AFEW}$ = 38.7/40.4	Exp-CLIP introduces a simple, learnable projection head that aligns the general vision–language feature space of a pre-trained model with a task-aware semantic space derived from an LLM encoder
Lan et al. [155]	ViT-L/14 (DINOv2; LLAVA 1.5)	$Ac{c}_{Affectnet}$ = 62.9; $Ac{c}_{RafDB}$ = 91.0; $Ac{c}_{ExpW}$= 68.1	EXPLLM leverages an AU model and GPT-4o to construct instruction–description data pairs that articulate the relationship between facial movements and emotion
Hu et al. [154]	LLaVA-LCS-558K (with LoRA)	$Ac{c}_{AffectNet}$ = 42.9; $Ac{c}_{RafDB}$ = 69.9; $Ac{c}_{ExpW}$= 68.1	FEALLM aligned descriptions of facial expressions (FE) and action units (AU), along with causal reasoning instructions to explain how AUs lead to FEs
Sălăgean et al. [156]	GPT-3.5	$Pre{c}_{CASMEII}$ = 96.0; $Re{c}_{CASMEII}$ = 93.0; $F{1}_{CASMEII}$ = 94	It integrates expert-defined rules and mechanisms for amplifying weak signals with the reasoning capabilities of a large language model

Table 10. Face expression recognition solutions that benefited from VLM captioning. In certain cases, the results are the consequence of specific conditions which are detailed in the cited references. $UAR$ = Unweighted Average Recall; $WAR$ = Weighted Average Recall.

Solution	VLM	Benchmark and Performance	Improvement
Zhao & Patras (2023) [159]	DFER-CLIP (CLIP + LLM text descriptions)	$UAR/WA{R}_{DFEW}$ = 59.6/71.3%; $UAR/WA{R}_{FERV39k}$ = 41.3/51.6%; $UAR/WA{R}_{MAFW}$ = 38.9/52.6%;	Uses ChatGPT-generated textual behavior descriptions instead of class labels → improved temporal FER generalization
Foteinopoulou & Patras (2024) [166]	EmoCLIP (CLIP + temporal transformer)	$UAR/WA{R}_{DFEW}$ = 58.1/62.1%; $UAR/WA{R}_{FERV39k}$ = 31.3/43.5%; $UAR/WA{R}_{MAFW}$ = 34.3/44.2%; $UAR/WA{R}_{AFEW}$ = 44.3/46.2%;	Uses sample-level captions of expressions and LLM-generated class descriptions for zero-shot video FER
Li et al. (2024) [15]	CLIPER (CLIP + METD)	$UAR/WA{R}_{FERV39k}$ = 41.2/51.4%; $WA{R}_{RAF-DB}$ = 91.8%	Learns multiple text prompts per emotion for richer semantics; better alignment without explicit captions
Chen et al. (2024) [160]	FineCLIPER (CLIP + AdaptERs + Video-LLaVA)	$UAR/WA{R}_{DFEW}$ = 66.0/76.2%; $UAR/WA{R}_{FERV39k}$ = 45.2/54.0%; $UAR/WA{R}_{MAFW}$ = 45.0/56.9%;	Fine-grained dynamic captions from Video-LLaVA improve temporal FER; surpasses CLIP baseline across datasets
Huang et al. (2025) [161]	Emotion-Qwen (Hybrid Expert Mixture)	$UAR/WA{R}_{DFEW}$ = 77.1/62.2%;	Integrates VER dataset for instruction fine-tuning with contextual and causal captions to enhance emotion reasoning
Seoh et al. [151]	Qwen2.5-VL; Aya Vision; InternVL2.5 8B-MPO	$Pre{c}_{FI}$ = 52.9; $Re{c}_{FI}$ = 52.2; $F{1}_{FI}$ = 48.5	EmoGist analyzes clusters of example images to pre-generate multiple context-specific descriptions via the VLM for emotion labels
Zhao et al. [153]	CLIP-L	$UAR/WA{R}_{RAF-DB}$ = 58.7/65.4; $UAR/WA{R}_{AffectNet}$ = 38.4/38.4; $UAR/WA{R}_{DFEW}$ = 40.2/47.1; $UAR/WA{R}_{AFEW}$ = 38.7/40.4	Exp-CLIP introduces a simple, learnable projection head that aligns the general vision–language feature space of a pre-trained model with a task-aware semantic space derived from an LLM encoder
Lan et al. [155]	ViT-L/14 (DINOv2; LLAVA 1.5)	$Ac{c}_{Affectnet}$ = 62.9; $Ac{c}_{RafDB}$ = 91.0; $Ac{c}_{ExpW}$= 68.1	EXPLLM leverages an AU model and GPT-4o to construct instruction–description data pairs that articulate the relationship between facial movements and emotion
Hu et al. [154]	LLaVA-LCS-558K (with LoRA)	$Ac{c}_{AffectNet}$ = 42.9; $Ac{c}_{RafDB}$ = 69.9; $Ac{c}_{ExpW}$= 68.1	FEALLM aligned descriptions of facial expressions (FE) and action units (AU), along with causal reasoning instructions to explain how AUs lead to FEs
Sălăgean et al. [156]	GPT-3.5	$Pre{c}_{CASMEII}$ = 96.0; $Re{c}_{CASMEII}$ = 93.0; $F{1}_{CASMEII}$ = 94	It integrates expert-defined rules and mechanisms for amplifying weak signals with the reasoning capabilities of a large language model

9. Discussion

9.1. Action Recognition

In human action recognition, leveraging captions translates into substantial performance gains; models can capture both high-level action semantics and fine-grained, frame-level nuances such as object interactions, hand movements, and subtle scene changes. Empirical results on benchmarks including MSR-VTT, YouCook2, Ego4D, and EPIC-KITCHENS demonstrate that VLMs using caption supervision consistently outperform unimodal approaches, particularly in zero-shot and few-shot scenarios where conventional video-only encoders struggle. These improvements are far from incremental—they significantly enhance both retrieval and generative capabilities, firmly establishing captions as a critical modality for advancing state-of-the-art HAR.

9.2. Violence Detection

Captioning and textual guidance enhance video anomaly and violence detection by providing a richer semantic context that complements visual information. By converting video content into structured textual representations, these approaches improve model interpretability, support fine-grained reasoning, and help capture subtle or complex anomalous behaviors. Overall, integrating textual cues consistently boosts detection performance and human-aligned understanding compared to relying on visual information alone. Despite these advances, current methods face key limitations: they can inherit biases from pre-trained LLMs/VLMs, rely on large datasets and heavy computation, and may propagate captioning errors or hallucinations. Many also struggle with long-term temporal dependencies and subtle behaviors, leading to missed or false detections, underscoring the need for more robust, efficient, and bias-aware textual–visual integration.

9.3. Emotion Recognition

Captioning and textual guidance enhance emotion recognition from both body language and facial expressions by adding semantic context that complements visual cues. Converting images into structured text improves efficiency and captures subtle behaviors, leading to more accurate emotion descriptions. Integrating textual cues consistently outperforms visual-only approaches in detection and human-aligned interpretation. However, current methods remain limited, often over-relying on assumed contextual correlations [167] and inheriting biases from the training data.

Beyond dataset bias and hallucination, a key limitation arises when visual cues themselves are unreliable; forged or manipulated images can mislead VLMs into generating confident but incorrect emotional attributions, while mental disorders (e.g., schizophrenia, autism spectrum conditions) may decouple facial or contextual signals from internal affect, creating systematic mismatches between expression and emotion. Context itself can also be deceptive—for instance, a smile in a funeral setting may not imply joy, but grief coping. These challenges highlight the fragility of context-based inference.

9.4. Issues and Mitigation

However, despite their promise, VLMs are not without limitations. Their reliance on web-scale data makes them prone to biases, and their generative capacities often lead to hallucinations or over-interpretations of visual signals [43,44]. In human state recognition, such errors may translate into misleading or biased predictions with serious downstream implications. Moreover, the exploration of large VLMs in the domain of context-aware recognition remains relatively under-developed, leaving open questions about optimal model choice, fine-tuning strategies, and dataset requirements.

Alternative mitigation solutions can be found in the VLM itself. VLMs, under specific prompts, are able to adjust their own biased predictions [167]. In other situations, specially designed solution are needed, and alternatives do exist.

To address such issues, several recent works have explored temporal and reasoning augmentations. Temporal modeling approaches add sequence encoders (e.g., Transformer layers on top of CLIP [131]) or specialized adapters to capture the dynamics of expressions across video frames. Reasoning strategies such as Chain-of-Thought prompting [148,168] guide VLMs to articulate intermediate steps, improving interpretability and robustness. Others propose fine-grained textual descriptions (e.g., multiple expression descriptors [15], LLM-generated prompts [159]) or hybrid multimodal fusion (e.g., combining landmarks, segmentation, and language adapters [160], or hybrid MoE compressors [161]). These methods expand beyond static frame-level prediction to incorporate temporal coherence, common-sense reasoning, and multi-cue integration, which, together, represent promising directions for reliable and fair emotion recognition.

10. Conclusions

Overall, the rise of Visual Language Models (VLMs) has had a positive impact in multiple areas related to the recognition and analysis of human states in images and video data. Their zero-shot capabilities and strong pre-training serve as the foundation for significant improvements across various domains. These solutions have become increasingly creative, inspiring alternative approaches—such as innovative VLM design choices, integration with complementary branches, and refined pre- and post-processing methods.

However, we argue that solution development is still in its golden age; the major improvements compared to non-VLM solutions are heavily emphasized, and the research community remains focused on enhancing key performance metrics. Nevertheless, we anticipate that this rapid progress will soon plateau, shifting attention toward fine-tuning details—specifically, mitigating biases and isolating spurious correlations. Ultimately, we hope this work provides key insights into the current state of the art and offers valuable suggestions for future developments in this area, where VLM-based captioning is becoming a central element driving further evolution.