Next Article in Journal
Estimating Weather Effects on Well-Being and Mobility with Multi-Source Longitudinal Data
Previous Article in Journal
A Network Scanning Organization Discovery Method Based on Graph Convolutional Neural Network
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Information
Volume 16
Issue 10
10.3390/info16100900
information-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

ADR: Attention Head Detection and Reweighting Enhance RAG Performance in a Positional-Encoding-Free Paradigm

by
Mingwei Wang
1,2,
Xiaobo Li
1,2,3,*,
Qian Zeng
1,2,
Xingbang Liu
1,2,
Minghao Yang
1,2 and
Zhichen Jia
1,2
1
AI Research Center, Research Institute of Petroleum Exploration and Development, PetroChina, Beijing 100083, China
2
Artificial Intelligence Technology R & D Center for Exploration and Development, China National Petroleum Corporation, Beijing 100083, China
3
National Key Laboratory for Multi-Resources Collaborative Green Production of Continental Shale Oil, Daqing 163712, China
*
Author to whom correspondence should be addressed.
Information 2025, 16(10), 900; https://doi.org/10.3390/info16100900
Submission received: 11 September 2025 / Revised: 4 October 2025 / Accepted: 10 October 2025 / Published: 15 October 2025
Browse Figures
Versions Notes

Abstract

Retrieval-augmented generation (RAG) has established a new search paradigm, in which large language models integrate external resources to compensate for their inherent knowledge limitations. However, limited context awareness reduces the performance of large language models in RAG tasks. Existing solutions incur additional time and memory overhead and depend on specific positional encodings. In this paper, we propose Attention Head Detection and Reweighting (ADR), a lightweight and general framework. Specifically, we employ a recognition task to identify RAG-suppressing heads that limit the model’s context awareness. We then reweight their outputs with learned coefficients to mitigate the influence of these RAG-suppressing heads. After training, the weights are fixed during inference, introducing no additional time overhead and remaining agnostic to the choice of positional embedding. Experiments on PetroAI further demonstrate that ADR enhances the context awareness of fine-tuned models.
Keywords: context awareness; large language model; retrieval-augmented generation; attention head context awareness; large language model; retrieval-augmented generation; attention head

Share and Cite

MDPI and ACS Style

Wang, M.; Li, X.; Zeng, Q.; Liu, X.; Yang, M.; Jia, Z. ADR: Attention Head Detection and Reweighting Enhance RAG Performance in a Positional-Encoding-Free Paradigm. Information 2025, 16, 900. https://doi.org/10.3390/info16100900

AMA Style

Wang M, Li X, Zeng Q, Liu X, Yang M, Jia Z. ADR: Attention Head Detection and Reweighting Enhance RAG Performance in a Positional-Encoding-Free Paradigm. Information. 2025; 16(10):900. https://doi.org/10.3390/info16100900

Chicago/Turabian Style

Wang, Mingwei, Xiaobo Li, Qian Zeng, Xingbang Liu, Minghao Yang, and Zhichen Jia. 2025. "ADR: Attention Head Detection and Reweighting Enhance RAG Performance in a Positional-Encoding-Free Paradigm" Information 16, no. 10: 900. https://doi.org/10.3390/info16100900

APA Style

Wang, M., Li, X., Zeng, Q., Liu, X., Yang, M., & Jia, Z. (2025). ADR: Attention Head Detection and Reweighting Enhance RAG Performance in a Positional-Encoding-Free Paradigm. Information, 16(10), 900. https://doi.org/10.3390/info16100900

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop