Rethinking Visual Attention for Reducing Hallucination in Large Vision–Language Models

Abstract

Large Vision–Language Models (LVLMs) have achieved strong performance in multimodal understanding and generation. However, they remain prone to hallucination, where generated content deviates from the visual input, reducing output reliability. We analyze the attention mechanism and identify two key issues in visual information use. The model exhibits insufficient overall attention to visual tokens and weak or dispersed attention to semantically relevant regions, limiting effective visual grounding. We propose a tuning-free attention intervention method applied at inference time. In the encoding stage, we apply a structured rescaling to the attention logits associated with visual tokens, introducing a structural bias in the visual subspace. In the decoding stage, we filter attention heads based on their response magnitudes and perform weighted aggregation using their global response intensities. This design reinforces salient visual evidence while suppressing weak or diffuse attention patterns. Experiments on CHAIR and POPE show that our method reduces hallucination without additional training. On the CHAIR benchmark, it reduces the sentence-level metric by 15.5% and the instance-level metric by 5.7% on average, while consistently improving performance across multiple LVLMs and maintaining strong results on general multimodal benchmarks such as MME.

1. Introduction

Large Vision–Language Models (LVLMs) [1,2,3,4,5] have achieved remarkable progress in multimodal understanding and generation tasks, demonstrating strong performance in applications such as image captioning, visual question answering, and multimodal dialogue [6,7,8,9]. Benefiting from the powerful semantic modeling and language generation capabilities of Large Language Models (LLMs) [10,11], as well as the effective visual representations from visual encoders [12], LVLMs are able to jointly process visual and textual information and perform cross-modal semantic alignment and reasoning in complex scenarios. As model size, training data, and instruction tuning continue to scale, LVLMs have achieved substantial improvements in both generation quality and interaction capabilities, showing great potential for real-world applications [13,14,15] and becoming a key research direction in multimodal artificial intelligence.

Despite these advances, LVLMs still suffer from hallucination [16,17,18,19,20,21] in practical applications, where the generated content is inconsistent with the actual visual input. This includes describing nonexistent objects, assigning incorrect attributes, or neglecting critical visual details while producing semantically plausible but inaccurate outputs. Previous research indicates that this issue is typically associated with the model’s over-reliance on language priors [22,23] during the decoding process, which weakens its ability to utilize genuine visual evidence. This issue not only reduces visual faithfulness and output reliability but also undermines user trust. Consequently, it limits the model’s deployment in high-reliability and safety-critical scenarios, such as medical analysis [24,25,26], autonomous driving [27,28], and decision support systems [29]. Therefore, mitigating hallucination while preserving generation capability is a critical challenge in vision–language research.

Existing approaches to hallucination mitigation can be broadly categorized into training-based methods and inference-time strategies. Prior work has improved cross-modal alignment by constructing high-quality instruction data [30,31,32], designing refined alignment objectives, or incorporating reinforcement learning from human feedback. While effective to some extent, these methods require additional data collection and training procedures, leading to higher computational cost and limited flexibility in adapting existing models. Existing works have explored contrastive decoding or visual constraint mechanisms to suppress hallucination [20,33,34]. The underlying rationale is to formulate distinct visual inputs or intervention signals and to contrast the output distributions across different conditions, ultimately penalizing generations that contradict the visual context. However, such approaches often rely on multiple forward passes or auxiliary models, resulting in higher inference complexity and reduced practicality for real-world deployment.

We propose a plug-and-play visual attention guidance method that directly modulates the model’s attention distribution at inference time, aiming to enhance the continuous utilization of visual information during generation. Specifically, in the encoding stage, we apply a structured rescaling to the attention logits associated with visual tokens, introducing a structural bias in the visual subspace. In the decoding phase, we further filter attention heads based on their response magnitudes and perform a weighted aggregation utilizing their global response intensities. This reinforces salient visual signals and suppresses weak or dispersed responses, improving the model’s focus on key regions. By implementing direct and fine-grained interventions on the attention mechanism, our method provides more stable visual constraints for the generation process, sustains the guidance of visual information, and effectively mitigates hallucination in multimodal generation.

The main contributions of this paper are summarized as follows:

• We analyze the visual information utilization patterns in LVLMs, observing a low proportion of visual attention and dispersed response distributions, which are intrinsically correlated with hallucination generation.
• We propose an inference-time attention intervention method that requires no fine-tuning. By structurally regulating the distribution of visual attention, together with a response-strength-based selection and weighted aggregation mechanism, the approach empirically improves the model’s utilization of visual information. It also offers an interpretable framework for understanding and mitigating hallucinations in vision–language models.
• Extensive experiments across multiple vision–language models and benchmark datasets demonstrate that the proposed method consistently reduces hallucination and improves the alignment between the generated results and the actual visual content and maintains high reasoning efficiency.

2. Related Work

2.1. Large Vision–Language Models

LLMs have achieved remarkable progress in natural language processing tasks, demonstrating strong capabilities in semantic modeling and reasoning [35,36]. Building upon this foundation, researchers have further incorporated visual information into language models to develop unified multimodal learning frameworks, enabling models to jointly process images and text and perform tasks such as visual question answering, image captioning, and multimodal dialogue.

Currently, most LVLMs adopt a vision encoder-language model architecture, where cross-modal alignment is achieved through intermediate bridging modules. One line of work leverages pre-trained visual encoders to extract image features, which are then projected into the embedding space of the language model via lightweight mapping modules. For instance, LLaVA-1.5 [37] employs CLIP ViT-L/14 [12] as the visual encoder and uses a linear projection layer to align visual features with language representations. Another line of work enhances the interaction between visual and textual modalities through explicit cross-modal interaction modules. For example, InstructBLIP [38] builds upon the BLIP-2 [39] framework by introducing an instruction-aware Q-Former, which selectively filters and reorganizes visual features to improve the effectiveness of vision–language interaction.

Despite the substantial progress achieved by these approaches in multimodal understanding and generation, the generated outputs may still fail to accurately reflect the visual content [16,17], manifesting as hallucinated objects, incorrect attributes, or descriptions inconsistent with the image semantics. This suggests that merely improving visual encoder capacity and cross-modal alignment is insufficient to ensure visual faithfulness during generation. Consequently, visual hallucination remains a critical challenge for current large vision–language models.

2.2. Hallucination in LVLMs

Hallucination in LLMs typically refers to the generation of content that lacks factual grounding or is semantically unreliable. This issue has been extensively studied from multiple perspectives, including data quality, training mechanisms, and inference processes. However, when models are extended to vision–language settings, hallucination is no longer limited to factual errors but also manifests as inconsistencies between generated content and the visual input, making the problem more challenging.

Hallucination in LVLMs typically manifests as incorrect perception or insufficient use of visual information. For example, the model may introduce nonexistent entities, assign incorrect attributes to existing objects, or produce descriptions that deviate from the overall semantics of the visual scene. Unlike in purely textual scenarios, such errors are often difficult to detect from the generated text alone, as the outputs may remain linguistically fluent and plausible while lacking correspondence with the visual input.

Existing studies suggest that hallucinations in LVLMs mainly arise from two factors. First, compared to visual encoders, language models typically possess stronger parameter capacity and semantic modeling ability. This cross-modal imbalance causes the model to rely excessively on language priors during generation, leading to an over-reliance on linguistic cues while underutilizing visual input [19,40,41,42]. In other words, the model tends to follow statistical patterns in language rather than strictly grounding its outputs in visual evidence. Second, current visual modules still exhibit limitations in modeling fine-grained visual semantics. Although visual encoders can effectively capture high-level semantic information, their representations of detailed aspects such as local attributes and complex relationships remain limited. This deficiency weakens the model’s understanding of the actual image content and increases the likelihood of errors, including incorrect attributes, missing details, and descriptions inconsistent with the visual input [17,19,34].

Despite the substantial progress achieved in multimodal understanding and generation, hallucination in LVLMs remains insufficiently addressed and continues to affect the reliability and visual faithfulness of model outputs. Effectively mitigating hallucination, therefore, remains a critical problem in vision–language research.

2.3. Approaches to Mitigate Hallucination in LVLMs

Existing studies have explored a variety of mitigation strategies, which can be broadly grouped into three categories.

One line of work improves vision–language alignment through additional training data [18,30,31,43,44,45,46]. These methods typically rely on high-quality annotations or human feedback and adopt techniques such as instruction tuning, preference optimization, or reinforcement learning to encourage better use of visual information during generation. Although effective to some extent, such approaches usually require extra data curation and additional training and thus face limited scalability for large models.

Another line of work seeks to alleviate hallucination by enhancing visual perception [45,47,48]. These methods often introduce auxiliary visual signals, such as depth maps, segmentation outputs, or other structured features, to strengthen the modeling of fine-grained details and local structures. While richer visual representations can reduce erroneous generations, the effectiveness of these methods often depends on the quality and availability of the auxiliary features, which may restrict their generalization across tasks and scenarios.

A third line of work addresses hallucination at inference time by intervening in the decoding process. These methods typically construct alternative visual inputs or adopt contrastive decoding schemes [17,20,34,49,50,51,52] using discrepancies in output distributions under different conditions to suppress visually inconsistent generations. Compared with training-based approaches, they offer greater flexibility by avoiding additional training. However, they often require multiple forward passes or may inadvertently weaken beneficial language priors, leading to trade-offs between inference efficiency and generation quality.

3. Understanding Attention in LVLMs

3.1. LVLM Decoding

Given an LVLM parameterized by $\omega$, it takes as input a textual sequence $X=(x_1,x_2,\dots ,x_N)$ and a sequence of visual features $V=(v_1,\dots ,v_{N_v})$. The two modalities are concatenated along the sequence dimension to form a unified representation $Z=[V;X]$, which is then fed into a Transformer with L layers to obtain contextualized representations.

During autoregressive decoding, the model generates output tokens step by step. At the t-th decoding step, the model conditions on the visual features V and the previously generated tokens $X_{<t}=(x_1,\dots ,x_{t-1})$. Let $Z_l^{\left(t\right)}$ denote the hidden representation at layer l and step t. The update at layer l can be formulated as:

$$Z_{l+1}^{\left(t\right)}=Z_l^{\left(t\right)}+\sum _{h=1}^H{\mathrm{Attention}}_{l,h}^{\left(t\right)}$$

(1)

where ${\mathrm{Attention}}_{l,h}^{\left(t\right)}(·)$ denotes the update contributed by the h-th attention head at layer l during step t. For simplicity, we omit the feed-forward network and layer normalization.

The next token is then sampled according to

$$x_t\sim p_{\omega }(x_t\mid V,X_{<t}).$$

(2)

3.2. Layer-Wise Attention Distribution

To analyze how LVLMs depend on different information sources during generation, we analyze the attention distribution in LLaVA-1.5 under the image captioning setting. Under this setup, we record the attention weights of all layers and heads and categorize tokens based on their sources, thereby characterizing how the model relies on visual and linguistic information across different layers.

At each decoding step t, we extract the attention weights from all layers and heads and group them according to token types (e.g., visual tokens, historical tokens, and system prompt tokens). Let $A_{l,h}^{\left(t\right)}\left(i\right)$ denote the attention weight assigned to the i-th token by the h-th attention head in the l-th layer at step t. For a set of tokens of type ${\mathcal{T}}_{\mathrm{type}}$, the aggregated attention strength at layer l is defined as:

$$S_{\mathrm{type}}\left(l\right)=\sum _t\sum _{h=1}^H\sum _{i\in {\mathcal{T}}_{\mathrm{type}}}{A}_{l,h}^{\left(t\right)}\left(i\right)$$

(3)

We further normalize the attention strengths across different token types to obtain their relative proportions at each layer:

$$R_{\mathrm{type}}\left(l\right)={\displaystyle \frac{S_{\mathrm{type}}\left(l\right)}{{\sum }_{{\mathrm{type}}^{\prime }}{S}_{{\mathrm{type}}^{\prime }}\left(l\right)}}$$

(4)

This metric captures the relative contribution of each token type at each layer and reflects how much the model relies on visual versus linguistic information.

As shown in Figure 1, system prompt tokens dominate attention across most layers, indicating that the model heavily relies on language priors during generation. Historical tokens maintain moderate attention in intermediate layers, reflecting the model’s ability to capture context. In contrast, visual tokens exhibit consistently lower attention proportions and noticeable fluctuations across layers.

Figure 1. Distribution of attention proportions for different token types across layers.

The attention proportion of visual tokens in the intermediate layers is relatively low. During the stage of progressive semantic formation, visual signals have limited influence on the generated outputs at this stage, which may cause the model to rely more on language priors and potentially yield outputs inconsistent with the image content. This phenomenon indirectly suggests that modulating the influence of visual information within the intermediate layers could help enhance the model’s capacity to utilize visual content.

3.3. Visual Attention Distribution

To further investigate the relationship between visual attention and hallucination, we analyze the attention distribution over visual tokens to examine how much the model relies on visual evidence when generating different types of tokens and how this relates to hallucination. Under this setup, we focus on the overall attention to visual information, along with its spatial distribution, for both correct and hallucinated tokens.

Let ${\mathcal{T}}_{\mathrm{img}}$ denote the set of all visual tokens. For a generated token $x_t$, we define its overall attention to visual information as:

$$G\left(x_t\right)=\sum _{l=1}^L\sum _{h=1}^H\sum _{i\in {\mathcal{T}}_{\mathrm{img}}}{A}_{l,h}^{\left(t\right)}\left(i\right)$$

(5)

here, $G\left(x_t\right)$ measures the overall dependence of the model on visual inputs when generating the current token. A larger value of $G\left(x_t\right)$ indicates that the model relies more on visual information, whereas a smaller value suggests that generation is more likely driven by language priors or contextual cues.

Based on this metric, we compare the attention distributions of correct tokens and hallucinated tokens. As shown in Figure 2, correct tokens typically exhibit attention that is concentrated on a small number of key visual locations, resulting in a sparse distribution with clear peaks. In contrast, hallucinated tokens tend to have weaker and more dispersed attention, lacking prominent concentration over specific visual regions. We hypothesize that when the model effectively utilizes visual evidence, its attention tends to concentrate on semantically relevant local regions, whereas in cases of insufficient visual dependency, attention is more likely to be dispersed.

Figure 2. Visual attention distribution over visual tokens for different types of generated tokens.

When attention is projected onto the image space, as illustrated in Figure 3, it can be observed that under the current experimental setup, when the model generates correct tokens, visual attention typically concentrates on a few semantically relevant regions, exhibiting strong response intensity and high spatial concentration. Conversely, when generating hallucinated tokens, the attention distribution is more dispersed, with weaker overall responses and a lack of distinct focal areas. Furthermore, when visual inputs are perturbed, attention responses gradually weaken and tend to scatter, reducing the model’s reliance on visual regions and prompting a shift towards a heavier dependence on language priors for generation.

Figure 3. Attention distributions for different types of generated tokens.

Taken together, these results suggest that, under our experimental setting, the model exhibits distinct differences in its utilization of visual information during generation. When visual attention demonstrates high response intensity and concentrates on semantically relevant regions, the generated results are typically consistent with the image content. Conversely, when visual attention is overall weak or dispersed, the model is more inclined to rely on language priors, thereby becoming more prone to producing outputs inconsistent with the visual content.

The above analysis is based on specific model and task settings. Thus, its conclusions should be interpreted as empirical observations rather than a universal characterization of LVLM behaviors. Nevertheless, these observations still provide valuable insights for our methodological design.

4. Methods

We observe two key issues in how the model uses visual information during generation in Section 3. First, overall attention to visual tokens is insufficient. Second, visual attention in semantically relevant regions is often diffuse and weak. These issues suggest that visual information fails to provide consistent and effective constraints during generation.

We propose an attention intervention strategy applied at inference time that directly modifies attention computation in a set of intermediate layers (e.g., layers 5–18). The proposed method can be viewed as a structural modification of the attention distribution, consisting of two complementary components. In the encoding stage, it rescales the attention logits of visual tokens. In the decoding stage, we refine attention by selecting and aggregating high-response visual regions, which improves the model’s use of visual information. The overall framework of our method is illustrated in Figure 4.

Figure 4. Overview of the proposed method. During the encoding stage, we structurally rescale the attention logits of visual tokens, while in the decoding stage, we filter, aggregate, and inject visual attention based on response intensity.

4.1. Visual Attention Biasing (VAB)

As shown in Section 3, visual tokens generally account for a small proportion of overall attention, suggesting that the model may underuse visual information during representation formation. We introduce a structural adjustment to the attention logits associated with visual tokens during the encoding stage, thereby introducing visual bias early in representation formation.

Let $Z_{l,h}\in {\mathbb{R}}^{T\times T}$ denote the attention logits of the h-th attention head in the l-th layer before the softmax normalization, where T is the length of the concatenated sequence of visual and textual tokens. Let $[s,e)$ denote the index range of visual tokens in the sequence. We apply a uniform rescaling to all attention logits that attend to visual tokens, yielding the modified logits:

$$Z_{l,h}^{\prime }=Z_{l,h}\odot M$$

(6)

where ⊙ denotes element-wise multiplication. The mask M is defined as

$$M(i,j)=\left\{\begin{array}{cc}\alpha ,\hfill & j\in [s,e)\hfill \\ 1,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(7)

where $\alpha >1$ is a scaling factor used to structurally rescale logits associated with visual tokens. Instead of applying pointwise or global amplification, it modifies the relative logit structure between visual and non-visual columns.

To analyze how this operation affects attention, consider a single row of logits corresponding to one query position. Let V denote the set of visual tokens. The total attention mass assigned to visual tokens after the softmax is

$$SV\left(\alpha \right)={\displaystyle \frac{{\sum }_{j\in V}{e}^{\alpha z_j}}{{\sum }_{j\in V}{e}^{\alpha z_j}+{\sum }_{k\notin V}{e}^{z_k}}}$$

(8)

its derivative with respect to $\alpha$ is

$${\displaystyle \frac{d{S}_V}{d\alpha }}=S_V\left(\alpha \right)\left(1-S_V\left(\alpha \right)\right)·{\displaystyle \frac{{\sum }_{j\in V}{z}_j{e}^{\alpha z_j}}{{\sum }_{j\in V}{e}^{\alpha z_j}}}.$$

(9)

The sign of ${\displaystyle \frac{d{S}_V}{d\alpha }}$ is determined by the term ${\displaystyle \frac{{\sum }_{j\in V}{z}_j{e}^{\alpha z_j}}{{\sum }_{j\in V}{e}^{\alpha z_j}}}$, which can be interpreted as an exponentially weighted average of the visual logits. Therefore, the effect of rescaling on attention mass assigned to visual tokens is not directly determined by the scaling factor, but depends on the distribution of the visual logits. In particular, this operation does not mean a monotonic increase in attention to visual tokens; rather, it alters the relative logit structure within the visual subspace, thereby inducing structural changes in attention.

Although the total attention mass assigned to visual tokens does not necessarily increase monotonically, the rescaling operation introduces a consistent structural effect within the visual subspace. Specifically, for any two visual tokens $a,b\in V$ with $z_a>z_b$, their relative contribution after the softmax is given by

$${\displaystyle \frac{e^{\alpha z_a}}{e^{\alpha z_b}}}=e^{\alpha (z_a-z_b)}.$$

(10)

when $\alpha >1$, the ratio is further amplified compared to the original $e^{z_a-z_b}$. The operation systematically enhances the relative differences among visual logits, thereby improving the distinction between high-response and low-response visual positions.

VAB amplifies relative differences within the visual subspace, thereby reshaping the attention distribution. As a result, attention is encouraged to concentrate on visually salient regions with stronger responses, while suppressing low-response or uncertain visual information. The design is consistent with the observations in Section 3, where visual attention often accounts for a small proportion and is widely dispersed. By enhancing the relative differences within the visual subspace, the method helps high-response visual signals maintain greater prominence during subsequent representation propagation.

As illustrated in the upper-right part of Figure 4, VAB is applied to the multi-head attention computation in several selected intermediate layers. Each small matrix in the figure represents the attention logits of a single attention head before the softmax, where the horizontal dimension corresponds to different source tokens and the vertical dimension corresponds to the current query position. The green regions denote columns associated with visual tokens, while the orange and blue regions correspond to text prompts and previously generated tokens, respectively. We rescale all logits that attend to visual tokens by applying a scaling factor $\alpha$ uniformly to the columns corresponding to the green regions, while leaving the remaining positions unchanged. The “Reweighting” module shown in the figure represents this process: the original attention logits are uniformly rescaled along the visual columns before being passed to the subsequent attention computation.

4.2. Response-Guided Attention Refinement (RAR)

After introducing a structural bias in the visual subspace during encoding, the effective utilization of visual evidence during autoregressive generation still depends on whether the model can consistently focus on semantically relevant visual regions. The analysis in Section 3.3 reveals that hallucinated tokens often correspond to a more dispersed visual response distribution. Consequently, we perform further filtering and aggregation on visual attention during the decoding phase to reinforce reliable visual signals.

At the t-th decoding step, let $Z_{l,h}^{\left(t\right)}$ denote the attention logits of the h-th attention head in the l-th layer before Softmax normalization. We extract the components corresponding to visual tokens, denoted as $z_{l,h}^{\left(t\right)}\in {\mathbb{R}}^{N_v}$, where $N_v$ is the number of visual tokens.

To characterize the response diversity across attention heads in the visual space, for each visual position j, we first select the top-K attention heads with the highest response magnitudes to form the core head set for this position, denoted as

$${\mathcal{H}}_l^{\left(t\right)}\left(j\right)={TopK}_h\left(\left|z_{l,h}^{\left(t\right)}\left(j\right)\right|\right),$$

(11)

where $|·|$ denotes the absolute value, used to measure the magnitude of the attention response. Because the responses of different attention heads at the same location may vary in sign, directly comparing raw values would confound sign information with response intensity, thereby underestimating the significance of large negative responses. Consequently, we utilize the absolute value as a unified metric for response intensity to more robustly characterize the strength of different attention heads. Furthermore, retaining the heads with larger response magnitudes helps identify attention patterns that exert a significant influence on the current visual location, thereby mitigating noise interference from weak responses.

To characterize the overall visual response intensity of different attention heads during the current generation step, we define a global reliability score for each head:

$$s_{l,h}^{\left(t\right)}={\displaystyle \frac{1}{N_v}}\sum _{j^{\prime }=1}^{N_v}\left|z_{l,h}^{\left(t\right)}\left(j^{\prime }\right)\right|$$

(12)

which reflects the overall response strength of the h-th attention head to visual information at the current layer.

For a given visual location j, we normalize the aforementioned scores exclusively within the core head set $H_l^{\left(t\right)}\left(j\right)$ to obtain the aggregation weights:

$${\omega }_{l,h}^{\left(t\right)}\left(j\right)={\displaystyle \frac{s_{l,h}^{\left(t\right)}}{{\sum }_{k\in H_l^{\left(t\right)}\left(j\right)}{s}_{l,k}^{\left(t\right)}}}$$

(13)

where $H_l^{\left(t\right)}\left(j\right)$ denotes the set of Top-K attention heads selected for visual location j in the l-th layer at the t-th generation step.

Based on these weights, we construct an aggregated representation at position j:

$$c_l^{\left(t\right)}\left(j\right)=\sum _{h\in H_l^{\left(t\right)}\left(j\right)}{\omega }_{l,h}^{\left(t\right)}\left(j\right)\left|z_{l,h}^{\left(t\right)}\left(j\right)\right|$$

(14)

Finally, the aggregated visual signal is injected back to the original attention logits:

$$z_{l,h}^{\left(t\right)}=z_{l,h}^{\left(t\right)}+\beta c_l^{\left(t\right)},$$

(15)

where $\beta$ is a hyperparameter controlling the strength of the injection. The updated visual logits are then combined with the non-visual components and passed through the subsequent Softmax operation to obtain the final attention distribution.

As illustrated in the bottom right panel of Figure 4, the responses of distinct attention heads across visual locations are unrolled into a matrix representation, where each row corresponds to the logits of an individual attention head over all visual tokens. For a given spatial location j, the figure color-codes the Top-K attention heads filtered by response magnitude, while the unselected heads remain unhighlighted. We observe that responses from different attention heads exhibit substantial variation at the same location. Subsequently, weighted aggregation is applied to these selected heads, consolidating the outputs of multiple high-response heads into a unified aggregated signal. Finally, this aggregated result is injected back into the original attention logits. This injection process fortifies the responses of critical visual regions while preserving the underlying architectural integrity.

This mechanism performs high-response filtering at each spatial location and conducts weighted aggregation based on global response intensity, thereby prioritizing visual signals with larger response magnitudes. Specifically, we use the response magnitude as a measure of the importance of each attention head, selecting and aggregating high-response heads to highlight key visual regions and enhance their influence during the decoding process.

4.3. Inference Strategy

To provide a clearer view of the proposed inference process, we summarize the overall procedure in Algorithm 1. The method integrates structural regulation of visual attention during encoding, where attention logits of visual tokens are rescaled to introduce a bias in the visual subspace, with response-guided refinement during decoding, where high-response attention heads are selected and aggregated based on their response intensities. It results in a cohesive and fine-grained attention control mechanism.

Algorithm 1: Overall Inference Procedure

5. Experiments

5.1. Evaluated LVLMs

We evaluate the proposed method on four representative 7B-scale vision–language models, including LLaVA-1.5 [37], InstructBLIP [38], Qwen-VL [53], and MiniGPT-4 [54]. These models differ significantly in visual encoders, cross-modal alignment strategies, and language model architectures, providing a comprehensive testbed to assess the applicability and generalization ability of our method across diverse LVLM designs.

Specifically, LLaVA-1.5 adopts a simple alignment strategy based on a visual encoder and a linear projection layer. InstructBLIP employs a Q-Former to enable more explicit cross-modal interaction. Qwen-VL incorporates a specially designed visual encoding and fusion mechanism, while MiniGPT-4 connects a pretrained visual encoder with a large language model through a lightweight alignment approach. Although these models demonstrate strong performance in multimodal understanding and generation tasks, they still exhibit varying degrees of hallucination, making them suitable benchmarks for evaluating the effectiveness of our method.

5.2. Baseline Methods

We compare the proposed method with several representative approaches for hallucination mitigation, including VCD [20], OPERA [55], SPIN [56], and DeCo [34]. These methods cover major approaches to inference-time hallucination mitigation, particularly those based on contrastive decoding and inference-time intervention, and thus provide strong and diverse baselines for evaluation.

5.3. Evaluation Benchmarks

5.3.1. CHAIR Benchmark

CHAIR [57] is a widely used benchmark for evaluating object hallucination in image captioning tasks. Built upon the MSCOCO dataset, it detects hallucinated objects by aligning model-generated captions with ground-truth annotations and identifying objects that are mentioned in the caption but absent from the image. The evaluation is based on two core metrics, ${\mathrm{CHAIR}}_S$ and ${\mathrm{CHAIR}}_I$, which characterize hallucination at the sentence level and instance level, respectively.

Formally, ${\mathrm{CHAIR}}_S$ measures the proportion of captions that contain at least one hallucinated object among all generated captions, while ${\mathrm{CHAIR}}_I$ measures the proportion of hallucinated objects among all mentioned objects. They are defined as

$${\mathrm{CHAIR}}_S={\displaystyle \frac{\left|\right\{\mathrm{captions}\mathrm{with}\mathrm{hallucinated}\mathrm{objects}\left\}\right|}{\left|\right\{\mathrm{all}\mathrm{captions}\left\}\right|}}$$

(16)

$${\mathrm{CHAIR}}_I={\displaystyle \frac{\left|\right\{\mathrm{hallucinated}\mathrm{objects}\left\}\right|}{\left|\right\{\mathrm{all}\mathrm{mentioned}\mathrm{objects}\left\}\right|}}$$

(17)

In our experiments, we sample 500 images from the MSCOCO 2014 validation set and use a unified prompt: “Please help me describe the image in detail.” to generate captions, ensuring a fair comparison across different methods.

5.3.2. POPE Benchmark

POPE [58] evaluates object-level hallucination of models by constructing a binary visual question answering task. Specifically, for each input image, candidate objects are selected from a predefined object set, and questions are formulated in the form “Is there a <object> in the image?”. The model is required to respond with “yes” or “no” for each question, indicating whether the queried object is present in the image.

The POPE benchmark comprises three data splits, namely Random, Popular, and Adversarial, which differ in how negative samples are constructed. In the Random split, negative objects are randomly sampled from those absent in the image. In the Popular split, negative samples are selected from a set of frequently occurring objects that are nonetheless missing from the image. In the Adversarial split, negative objects are preferentially chosen based on their co-occurrence relationships with objects present in the image, despite being absent, making the evaluation more challenging. The benchmark is built upon three datasets, MSCOCO, A-OKVQA, and GQA. For each split, 500 images are selected from each dataset, and six questions are constructed per image, resulting in a total of 27,000 question–answer pairs.

During evaluation, model responses are first converted to binary labels (“yes” or “no”) and then compared with the ground truth. The evaluation metrics include Accuracy and F1-score. Accuracy is defined as the proportion of correctly predicted instances among all questions. F1-score considers both precision and recall, providing a more comprehensive assessment of model performance under class imbalance. All experiments follow the official POPE evaluation protocol and are conducted using the publicly released data splits.

5.3.3. MME Benchmark

MME [59] is a comprehensive benchmark for evaluating multimodal large language models in terms of visual perception and cognitive capabilities. It covers a wide range of visual understanding tasks, including object recognition, attribute identification, text recognition, numerical reasoning, and spatial relationship comprehension. The benchmark consists of ten perception-level subtasks and four cognition-level subtasks and evaluates models through diverse visual question-answering formats.

During evaluation, each subtask is composed of a set of image–question pairs. The model is required to generate textual responses based on the input images. The outputs are then compared with ground-truth answers and scored according to their correctness. Scores are first computed for each individual subtask and then aggregated to obtain an overall performance metric, reflecting model capabilities across different aspects of visual understanding.

In terms of evaluation metrics, we follow the official MME protocol and adopt the subtask score as the primary metric. For each subtask, the score is defined as the sum of Accuracy and Accuracy+, where Accuracy measures the average correctness over all image–question pairs, and Accuracy+ requires all questions associated with the same image to be answered correctly. The final performance is obtained by summing the scores across all subtasks.

5.4. Implementation Details

Due to the differences among LVLMs in the number of visual tokens and cross-modal fusion mechanisms, we adopt a unified set of hyperparameters, with minor adjustments to a few model-specific components. Unless otherwise specified, our method is applied to the intermediate layers (layers 5–18) to avoid insufficient visual feature integration in shallow layers and the dominance of strong language priors in deeper layers.

Across all models, we set the visual attention scaling factor $\alpha =1.3$ and the attention injection strength $\beta =0.5$. During the response-based filtering stage, we retain the $K=24$ attention heads with the strongest responses for LLaVA-1.5, Qwen-VL, and MiniGPT-4. For InstructBLIP, due to the presence of visual tokens, we use $K=12$. All hyperparameters are fixed across datasets without additional tuning to evaluate the stability and generalization ability of the proposed method across models and tasks.

All methods are systematically evaluated under a unified experimental setting. We fix the random seed to 42 and adopt a consistent decoding strategy across all methods. For all baseline approaches, we rely on their official implementations or publicly available codebases, following the recommended parameter configurations. All experiments are conducted on a computing environment equipped with NVIDIA RTX 3090 GPUs.

5.5. Experiment Results

5.5.1. Results on CHAIR Benchmark

Table 1 presents the evaluation results of different LVLMs on the CHAIR benchmark under the setting of max new tokens = 512. Overall, the proposed method achieves the best or near-best performance across four models and three decoding strategies. Compared to the Vanilla baseline, our method reduces ${\mathrm{CHAIR}}_S$ by 15.5% and ${\mathrm{CHAIR}}_I$ by 5.7% on average across all models and decoding strategies, demonstrating its effectiveness in mitigating both sentence-level and instance-level hallucinations.

Table 1. Evaluation results of four LVLMs on the CHAIR benchmark under different decoding strategies. ↓ indicates lower is better. Bold indicates the best result.

Decoding	Method	LLaVA-1.5		Qwen-VL		InstructBLIP		MiniGPT-4
		${\mathbf{CHAIR}}_{\mathit{S}}$ ↓	${\mathbf{CHAIR}}_{\mathit{I}}$ ↓	${\mathbf{CHAIR}}_{\mathit{S}}$ ↓	${\mathbf{CHAIR}}_{\mathit{I}}$ ↓	${\mathbf{CHAIR}}_{\mathit{S}}$ ↓	${\mathbf{CHAIR}}_{\mathit{I}}$ ↓	${\mathbf{CHAIR}}_{\mathit{S}}$ ↓	${\mathbf{CHAIR}}_{\mathit{I}}$ ↓
Greedy	Vanilla	47.4	13.2	45.4	12.1	57.3	23.9	31.8	10.7
	DeCo	37.8	11.1	42.2	10.7	41.2	14.4	27.0	8.8
	Ours	30.0	7.2	33.2	9.9	33.8	9.4	26.6	7.2
Beam Search	Vanilla	52.2	14.4	41.2	10.4	56.4	16.2	33.9	10.7
	OPERA	44.6	12.8	34.6	9.5	46.4	14.2	26.2	9.5
	DeCo	33.0	9.7	32.0	8.7	43.8	12.7	24.8	7.5
	Ours	29.2	9.5	23.6	8.0	34.8	9.8	22.8	7.3
Sampling	Vanilla	55.0	16.5	47.0	12.9	56.2	25.6	33.6	11.2
	VCD	52.8	15.3	46.4	11.9	58.0	17.0	33.8	11.1
	SPIN	35.6	13.9	35.4	11.9	48.8	14.0	27.0	9.0
	DeCo	42.0	11.6	43.8	11.8	43.6	12.9	30.8	9.5
	Ours	35.4	9.2	35.2	10.0	40.6	13.6	26.2	8.0

In terms of decoding strategies, the proposed method consistently outperforms the baselines under Greedy, Beam Search, and Sampling, indicating strong adaptability to different decoding schemes. Across models, consistent improvements are observed on LLaVA-1.5, Qwen-VL, InstructBLIP, and MiniGPT-4, further validating the generalization capability of our method across diverse LVLM architectures.

During practical inference and evaluation, we observe that MiniGPT-4 not only incurs significantly longer inference times than other models but also shows the smallest overall performance gains. This is due to the combined effects of its architecture and generation mechanism. MiniGPT-4 is equipped with an extremely large billion-parameter-scale visual encoder, and its engineering implementation typically relies on 8-bit quantized loading with relatively low computational efficiency, which constitutes a fundamental bottleneck for its inference speed. From a statistical perspective, its relatively low initial hallucination baseline further constrains the absolute margin for performance improvement.

To further evaluate the stability of the proposed method in long-sequence generation, we conduct experiments on LLaVA-1.5 by setting max new tokens to 64, 128, and 256, respectively, and perform comparisons under the same configurations. As shown in Table 2, the overall hallucination levels of all methods increase with longer generation lengths. Nevertheless, the proposed method consistently achieves the lowest ${\mathrm{CHAIR}}_S$ and ${\mathrm{CHAIR}}_I$ scores across all settings, demonstrating its effectiveness in mitigating hallucinations even in long-sequence generation scenarios.

Table 2. CHAIR evaluation results on LLaVA-1.5 under different generation length settings with a sampling-based decoding strategy. Bold indicates the best result.

Method	Max New Token 64		Max New Token 128		Max New Token 256
	${\mathbf{CHAIR}}_{\mathit{S}}$ ↓	${\mathbf{CHAIR}}_{\mathit{I}}$ ↓	${\mathbf{CHAIR}}_{\mathit{S}}$ ↓	${\mathbf{CHAIR}}_{\mathit{I}}$ ↓	${\mathbf{CHAIR}}_{\mathit{S}}$ ↓	${\mathbf{CHAIR}}_{\mathit{I}}$ ↓
Vanilla	23.4	7.1	53.6	15.3	55.0	16.5
VCD	23.2	7.1	55.0	15.1	52.8	15.3
SPIN	24.8	8.8	32.6	12.0	34.5	12.7
DeCo	19.4	6.3	41.0	11.8	42.0	11.6
Ours	18.4	6.0	31.8	9.3	32.3	9.6

5.5.2. Results on POPE Benchmark

Table 3 presents the evaluation results under three POPE settings. Overall, the proposed method achieves consistent improvements on both LLaVA-1.5 and Qwen-VL, outperforming the Vanilla baseline in terms of both Accuracy and F1-score. Specifically, on LLaVA-1.5, our method yields an average improvement of 5.86% in Accuracy and 5.08% in F1-score across the three settings. On Qwen-VL, the corresponding gains are 2.85% and 3.41%, respectively. These results demonstrate that the proposed approach effectively mitigates object-level hallucinations and enhances the model’s ability to accurately determine the presence of target objects.

Table 3. Evaluation results on the POPE benchmark under different settings for LLaVA-1.5 and Qwen-VL with a sampling-based decoding strategy. Bold indicates the best result.

Model	Method	Random		Popular		Adversarial
		Accuracy ↑	F1-Score ↑	Accuracy ↑	F1-Score ↑	Accuracy ↑	F1-Score ↑
LLaVA-1.5	Vanilla	82.42	82.70	76.57	79.56	71.09	72.58
	VCD	84.69	85.85	77.37	80.57	71.13	76.37
	SPIN	86.29	84.80	81.81	80.47	80.31	79.47
	Ours	86.84	86.72	82.76	83.42	78.06	79.94
Qwen-VL	Vanilla	85.58	84.01	83.72	82.47	80.84	79.94
	VCD	87.32	86.19	85.24	84.29	82.27	81.67
	SPIN	88.49	88.22	84.66	84.86	80.49	81.48
	Ours	88.93	88.06	86.12	85.28	83.63	83.30

Across different difficulty settings, our method demonstrates strong competitiveness on both models under the Random and Popular splits, achieving performance that is superior or close to the best in most cases. Under the more challenging Adversarial split, although some competing methods perform comparably on certain metrics, our approach maintains advantages across the majority of metrics and yields particularly strong results on Qwen-VL. These findings indicate that the proposed method remains robust when handling highly distracting queries.

5.5.3. Results on MME Benchmark

Table 4 presents the results on the MME benchmark. Our method achieves consistent performance improvements on both LLaVA-1.5 and Qwen-VL, outperforming the Vanilla baseline on the aggregated metrics of Perception and Recognition. These results indicate that the proposed approach effectively reduces hallucinations without compromising the model’s inherent visual understanding capabilities.

Table 4. Evaluation results on the MME benchmark for LLaVA-1.5 and Qwen-VL with a sampling-based decoding strategy. ↑ indicates higher is better. Bold indicates the best result.

Model	Decoding	Perception ↑	Recognition ↑	Total ↑
LLaVA-1.5	Vanilla	1254.77	271.79	1526.56
	VCD	1363.96	353.29	1717.25
	SPIN	1294.00	273.60	1567.60
	Ours	1338.71	365.36	1719.07
Qwen-VL	Vanilla	1364.74	320.36	1685.10
	VCD	1403.17	317.89	1721.06
	SPIN	1386.52	315.91	1702.43
	Ours	1420.37	343.57	1763.94

Figure 5 presents the fine-grained results of LLaVA-1.5 across different subtasks. As observed, the proposed method exhibits consistent improvements across both perception-oriented tasks and recognition-oriented tasks. This indicates that by strengthening visual constraints through controlled modulation, the model can not only reduce hallucinations but also improve the alignment between visual information and language reasoning, thereby enhancing overall multimodal performance.

Figure 5. Performance comparison across different sub-tasks of the MME benchmark with LLaVA-1.5.

Figure 6 presents the fine-grained results across different subtasks on Qwen-VL. We observe that, for most perception-oriented tasks (such as Color, Posters, and Scene), the proposed method brings notable performance improvements. For recognition-oriented tasks, certain subtasks (e.g., Numerical Calculation and Commonsense Reasoning) also show moderate gains, while changes on tasks such as Code Reasoning remain relatively limited. These results indicate that the benefits of the proposed method are primarily reflected in visual-related capabilities while also providing auxiliary improvements for some language reasoning tasks. Overall, the proposed approach demonstrates a consistent trend of performance improvement on Qwen-VL, further validating its effectiveness across different models.

Figure 6. Performance comparison across different sub-tasks of the MME benchmark with Qwen-VL.

5.6. Ablation Study

We conduct ablation studies on the CHAIR benchmark under a sampling-based decoding strategy.

5.6.1. Module Ablation Study

Table 5 presents the individual contributions of the VAB and RAR modules, as well as their combined effect. Across LLaVA-1.5, Qwen-VL, and InstructBLIP, both modules independently yield consistent improvements, while their joint application achieves the best performance. This suggests that the structural regulation of visual attention during the encoding stage and the response-based dynamic refinement mechanism exhibit strong complementarity.

Table 5. Ablation results of different modules (VAB and RAR) across four vision–language models. Bold indicates the best result.

Method	LLaVA-1.5		Qwen-VL		InstructBLIP		MiniGPT-4
	${\mathbf{CHAIR}}_{\mathit{S}}$ ↓	${\mathbf{CHAIR}}_{\mathit{I}}$ ↓	${\mathbf{CHAIR}}_{\mathit{S}}$ ↓	${\mathbf{CHAIR}}_{\mathit{I}}$ ↓	${\mathbf{CHAIR}}_{\mathit{S}}$ ↓	${\mathbf{CHAIR}}_{\mathit{I}}$ ↓	${\mathbf{CHAIR}}_{\mathit{S}}$ ↓	${\mathbf{CHAIR}}_{\mathit{I}}$ ↓
Vanilla	55.0	16.5	47.0	12.9	56.2	25.6	33.6	11.2
w/VAB	46.4	14.2	41.8	12.2	49.4	15.0	30.4	9.5
w/RAR	40.0	10.5	36.8	10.4	50.2	15.1	24.8	9.4
w/VAB + RAR	35.4	9.2	35.2	10.0	40.6	13.6	26.2	8.0

Taking LLaVA-1.5 as an example, after introducing VAB, ${\mathrm{CHAIR}}_S$/${\mathrm{CHAIR}}_I$ decreases from 55.0/16.5 to 46.4/14.2; using RAR alone further reduces them to 40.0/10.5; when both VAB and RAR are applied, the model achieves the best performance of 35.4/9.2. Similar trends are observed on Qwen-VL and InstructBLIP.

Although both VAB and RAR individually improve performance on MiniGPT-4, their combined effect does not match the optimal results observed in other models. This phenomenon may be attributed to the visual fusion mechanism and attention distribution characteristics of MiniGPT-4, where multiple interventions fail to produce fully additive effects.

5.6.2. Parameter Sensitivity Analysis

To validate the robustness of the proposed method with respect to key hyperparameters, we further conduct an analysis on LLaVA-1.5, examining the effects of the visual scaling factor $\alpha$, the number of attention heads k, and the attention injection strength $\beta$ on model performance. The results are shown in Figure 7. The first row corresponds to the ${\mathrm{CHAIR}}_I$ metric, where lower values indicate fewer hallucinations; the second row reports the F1 score, where higher values indicate better performance.

Figure 7. Parameter sensitivity analysis under different hyperparameter settings based on LLaVA-1.5.

The results show that when $\alpha$ and k vary within a certain range, the overall model performance remains relatively stable, indicating that the proposed method is robust to these two hyperparameters. Specifically, a moderate $\alpha$ helps enhance the contribution of visual information in the attention mechanism, while an appropriate k more effectively selects attention heads with stronger responses, thereby retaining more reliable attention patterns. In contrast, $\beta$ has a more pronounced impact on performance. As $\beta$ increases, ${\mathrm{CHAIR}}_I$ consistently decreases, suggesting that stronger aggregation signals help suppress hallucination. However, when $\beta$ becomes excessively large, the F1 score drops significantly, indicating that overly strong intervention negatively affects the model’s normal generation capability. This suggests that $\beta$ plays a trade-off role between hallucination suppression and generation quality.

Overall, the proposed method maintains stable performance over a relatively wide range of $\alpha$ and k, while $\beta$ needs to be carefully tuned within a reasonable range to balance hallucination reduction and generation performance.

5.6.3. Layer Range Analysis

Due to the large combinatorial space of layer ranges, exhaustively searching over all possible intervals would incur substantial computational cost. Therefore, instead of performing a comprehensive traversal of all layer combinations, we constrain the search space based on the analysis presented in Section 3.2.

As illustrated in Figure 1, visual tokens exhibit relatively low attention weights in the middle layers, which correspond to a critical stage where cross-modal semantics are progressively formed. As illustrated in Figure 8, we therefore focus the intervention range primarily on the middle layers and conduct ablation studies within this region.

Figure 8. Ablation results on different intervention layer ranges based on LLaVA-1.5.

For LVLMs with different fusion designs, we do not assume the existence of a universally optimal layer interval. Rather than treating the 5–18 range as a fixed rule, we emphasize the underlying phenomenon revealed in Section 3.2. Accordingly, our method focuses on applying interventions to the middle layers.

5.6.4. Efficiency Analysis

To evaluate the practical efficiency of the proposed method, we further compare the inference speed and memory consumption of different approaches on LLaVA-1.5. The results are illustrated in Figure 9.

Figure 9. Efficiency comparison of different methods on LLaVA-1.5. The horizontal axis represents generation speed, the vertical axis denotes the ${\mathrm{CHAIR}}_I$ metric, and the bubble size indicates memory consumption. The proposed method achieves a favorable balance between inference speed and memory usage.

As shown in Figure 9, the Vanilla method achieves the highest inference speed but suffers from relatively severe hallucinations. OPERA incurs substantial overhead in both memory and computation while failing to deliver satisfactory performance. In comparison, VCD, DeCo, and SPIN reduce hallucinations to some extent, but at the cost of decreased inference speed. In contrast, our method does not introduce additional forward passes and is able to further reduce the ${\mathrm{CHAIR}}_I$ metric while maintaining high inference speed and low memory consumption, demonstrating a more favorable efficiency-performance trade-off.

To provide a more precise quantitative comparison, Table 6 reports the inference speed and GPU memory consumption of each method, complementing the efficiency comparison presented in Figure 9.

Table 6. Comparison of inference efficiency of different methods on LLaVA-1.5.

Method	Memory (MB)	TPS
Vanilla	15,401	32.7
OPERA	22,177	4.6
SPIN	15,501	30.8
VCD	15,789	15.5
DeCo	15,529	27.3
Ours	15,579	30.5

5.6.5. Stability Analysis

To evaluate the stability of our method under sampling decoding, we conduct experiments with multiple random seeds using sampling-based decoding. We consider two representative models, LLaVA-1.5 and Qwen-VL, and run each experiment with three different random seeds (51, 123, and 2026), corresponding to s1–s3 in Table 7. The table reports both the seed-wise results and their mean and standard deviation.

Table 7. Stability analysis on the CHAIR benchmark under sampling-based decoding with LLaVA-1.5 and Qwen-VL. Bold indicates the best result.

Metric	Method	LLaVA-1.5				Qwen-VL
		s1	s2	s3	Mean ± Std	s1	s2	s3	Mean ± Std
${\mathrm{CHAIR}}_S$ ↓	Vanilla	53.4	55.2	52.6	53.73 ± 1.09	48.1	45.6	46.2	46.63 ± 1.06
	Ours	34.0	32.8	34.6	33.80 ± 0.75	35.7	34.8	36.0	35.50 ± 0.51
${\mathrm{CHAIR}}_I$ ↓	Vanilla	16.1	15.8	15.3	15.73 ± 0.33	12.6	12.1	13.0	12.57 ± 0.37
	Ours	9.5	8.9	9.1	9.17 ± 0.25	10.1	9.8	10.4	10.10 ± 0.24

As shown in Table 7, the baseline models exhibit noticeable performance variations across different random seeds, indicating sensitivity to the sampling decoding process. While our method is also affected by randomness, it consistently achieves lower standard deviations compared to the baseline, demonstrating reduced variance and improved stability. Meanwhile, our method maintains clear performance gains, indicating stronger robustness under sampling decoding.

5.7. Case Study

5.7.1. Successful Case

To further analyze how the proposed method operates in specific scenarios, we present a representative example from the image captioning task, as illustrated in Figure 10.

Figure 10. Case study on LLaVA-1.5. Compared with the Vanilla baseline, the proposed method reduces hallucinated objects (highlighted in red) in image captioning and improves accuracy in object existence prediction.

The image depicts a man flying a kite on a beach. While the baseline model is able to identify the main elements (e.g., the person and the kite), it also generates content that is inconsistent with the image, such as hallucinated objects like “nine people” and “backpack.” This suggests that the model fails to sufficiently ground its generation in the visual information, leading to the introduction of content that does not correspond to the image.

After introducing the proposed method, the model’s decision-making process changes accordingly. First, during the encoding stage, the method structurally regulates the attention logits associated with visual tokens, leading to a more pronounced bias in the visual attention distribution during representation formation. This, in turn, makes key visual information more distinguishable in subsequent representations. Then, during decoding, the model tends to preserve high-response regions that are relevant to the current visual content at each generation step, thereby continuously reinforcing their contribution throughout the generation process.

As a result of this adjustment, the model progressively prioritizes semantically grounded content (e.g., the kite and the person) while suppressing candidates lacking visual support (e.g., “backpack”). Consequently, the final generated output better aligns with the image content, effectively reducing hallucination.

5.7.2. Failure Case

As shown in Figure 11, this example illustrates a scene where a group of children is playing soccer on a grassy field, representing a typical failure case of the proposed method. Both the Vanilla and the proposed method are able to correctly capture the overall semantics, such as recognizing that multiple children are engaged in a soccer activity.

Figure 11. Failure case based on LLaVA-1.5. Tokens highlighted in red indicate hallucinations.

Both methods exhibit noticeable deviations in fine-grained details. In terms of quantity estimation, neither the baseline nor the proposed method provides accurate results, generating descriptions such as “nine children” that do not match the actual scene. This can be attributed to occlusion among objects and their relatively dispersed spatial distribution, which makes it difficult for the model to derive reliable counting cues from visual information. In addition, the proposed method misinterprets the action relationship, incorrectly identifying the interacting subject with the ball as “a boy in a yellow vest,” which is not fully consistent with the visual content.

These results indicate that, although the model is capable of capturing the overall scene semantics, it still suffers from limitations in precise counting and fine-grained action understanding, particularly in scenarios involving occlusion or complex multi-object distributions.

6. Discussion

The core of the proposed method is to guide the model to use visual information more consistently during inference by structurally modifying the attention distribution. Unlike approaches that rely on additional forward passes or auxiliary models, our method operates directly on attention computation to influence the model’s generation decisions. During encoding, a structural bias is introduced in the visual subspace, helping preserve clearer distinctions among visual signals as they propagate across layers. During decoding, the response-guided refinement mechanism selects and aggregates visual regions at each generation step, making high-response areas more prominent in the attention distribution. This combination of “global bias and local selection” steers the model toward generating content grounded in visual evidence, thereby reducing outputs that lack visual support.

Despite these advantages, the proposed method still has limitations in complex scenarios. For instance, in cases involving densely distributed multiple objects or occlusions, the model may struggle with precise counting and fine-grained relation understanding. Such failures often result from uncertainty in visual evidence: when clear visual cues are lacking, even with attention modulation and greater emphasis on high-response visual regions, the model may still fail to extract reliable information. In addition, in scenarios involving complex actions or semantic ambiguity, attention responses may be distributed across multiple candidate regions, leading to incorrect relation reasoning. These observations suggest that while the method improves the use of visual information in practice, its effectiveness remains limited when the visual input itself is ambiguous or insufficient.

Additionally, the performance of the proposed method varies across vision–language models due to differences in their cross-modal fusion mechanisms. As observed from the experimental results, the proposed method yields relatively stable improvements on models such as LLaVA-1.5 and Qwen-VL, whereas exceptions exist in certain models (e.g., MiniGPT-4). We hypothesize that this divergence mainly results from structural differences in the fusion pathways of visual and linguistic information among the models. When visual information directly participates in generation through the attention mechanism, modulating the attention distribution is more effective. Conversely, in models where the injection of visual information is relatively indirect, the effectiveness of this intervention strategy may be limited.

7. Conclusions

This paper investigates hallucination in large vision–language models from the perspective of attention mechanisms and proposes an attention intervention method applied during the inference stage. Through an analysis of attention distributions, we observe that insufficient attention to visual tokens and dispersed visual responses weaken the constraints provided by visual information during generation. Based on these findings, we design a visual attention guidance strategy consisting of VAB and RAR, which improves the model’s utilization of visual information in two ways: visual reweighting and response-driven selection and aggregation.

Experimental results across multiple vision–language models and evaluation benchmarks demonstrate that the proposed method can consistently reduce hallucinations while maintaining strong visual understanding. This suggests that regulating attention distributions during inference can improve the visual consistency and reliability of LVLM outputs.

The proposed method still has limitations. For example, the current approach is validated on image–text scenarios, and its applicability to more complex multimodal tasks requires further investigation. In addition, this work focuses on the attention mechanism and does not analyze the role of other internal components in the formation of hallucinations.

Future work can explore several directions. First, more adaptive attention regulation strategies can be developed to reduce the need for manual hyperparameter tuning. Second, the method can be extended to more complex multimodal tasks. Third, further analysis of different internal mechanisms in hallucination is needed. Overall, this work provides a useful perspective on understanding and mitigating hallucination in LVLMs through the lens of attention mechanisms.