Game-Theoretic Multi-LLM Collaboration for Attribute-Aware Open-Vocabulary Object Detection

Abstract

Open-vocabulary object detection (OVD) fails at attribute-level discrimination: when instances share a class label yet differ in color, material, or texture, category names provide no appearance-specific cues. Prior attempts to enrich text inputs with LLM-generated descriptions are limited by single-model distribution bias, producing coverage gaps and unstable attribute quality. We propose a Concept Expander framework built on cooperative multi-LLM game theory. Three heterogeneous LLMs generate candidate attributes in parallel; a cooperative Nash equilibrium then selects the final subset by maximizing each model’s minimum utility gain, jointly enforcing semantic quality and cross-source diversity without amplifying any single model’s bias. The resulting Concept Repository contains approximately 5000 discriminative visual priors. A lightweight retrieval module injects the top-k matched attributes into region-level visual features via residual fusion, preserving CLIP’s pretrained alignment while enriching instance representations with fine-grained semantic priors. A semantic consistency loss anchors enhanced features to ground-truth class semantics throughout training. On LVIS, rare-category APr rises from 22.2 to 28.5; on RefCOCO, attribute-conditioned localization accuracy reaches 54.8, confirming that structured multi-LLM semantic priors improve discrimination across long-tail and high-confusion benchmarks.

1. Introduction

Traditional object detection relies on the closed-set assumption that the training and test categories are the same, learning a mapping from visual features to predefined labels using a large number of annotated samples. However, real-world applications continue to introduce new concepts, such as autonomous driving encountering unseen traffic signs, industrial inspection facing new types of defects, and ecological monitoring requiring the recognition of rare species. Progress in vision–language pre-trained models provides a new opportunity to address this problem. CLIP [1] is trained on 400 million image–text pairs with contrastive learning and builds a shared semantic space for images and text, allowing the model to understand arbitrary textual descriptions and match them with visual content. Building on this idea, open-vocabulary object detection (OVD) reformulates detection as a vision–language alignment task and can perform zero-shot detection by only providing text descriptions of new categories. Early works such as ViLD [2] transfer CLIP’s capability to region-level detection through knowledge distillation; RegionCLIP [3] directly trains a region encoder using pseudo-labeled region–text pairs; and OVR-CNN [4] and Detic [5] expand the detection vocabulary with image classification data, laying the technical foundation for this research area.

Building on this foundation, OVD has developed a diverse technical landscape. Transformer-based end-to-end frameworks have become the dominant paradigm: GLIP [6] and GLIPv2 [7] pre-train on large-scale data with a unified framework for localization and vision–language understanding; Grounding DINO [8] and its upgraded version [9] introduce joint text–image encoding and feature enhancement mechanisms; and CORA [10] improves CLIP adaptation through region prompts and anchor-box pre-matching. To better utilize unlabeled data, self-training strategies have been widely explored: DST-Det [11] proposes a dynamic self-training [12] mechanism; Taming Self-training systematically studies pseudo-label noise handling; and PromptDet and Learning to Prompt [13] optimize text representations through prompt learning [14]. Generative and real-time directions have also made progress: DetCLIPv2 [15] and DetCLIPv3 [16] build scalable generative detection frameworks, and YOLO-World [17] achieves OVD while maintaining real-time performance. These works report clear improvements on standard benchmarks such as LVIS [18] and COCO [19]. Notably, recent studies focus on expanding the detection vocabulary size: V3Det [20] includes 13,204 categories, and OmniLabel [21] builds a language-driven, diverse evaluation set, reflecting the field’s continuous emphasis on the “number of categories”.

However, the above methods reveal a key limitation of category-level detection: performance can degrade markedly when the task requires separating visually similar instances with different attributes. Although recent studies have scaled up the vocabulary size (e.g., V3Det with more than ten thousand categories), this primarily broadens recognition coverage while paying limited attention to intra-class semantic granularity. Bianchi et al. show through systematic evaluation that existing OVD detectors fall short on attribute-aware understanding [22]; in scenarios that require distinguishing subtle differences such as “red chair” versus “blue chair,” accuracy is substantially lower than that of conventional detection. OVDEval [23], proposed by Yao et al., evaluates performance along semantic dimensions and finds that subtasks involving visual attributes—such as color, material, and spatial relations—exhibit an AP50 drop of more than 10 percentage points relative to the overall level. Bravo et al. further point out that models struggle to learn reliable correspondences between visual features and attributes such as appearance priors, object states, and functional properties [24]. Although several targeted methods have emerged recently—GUIDED improves recognition and discrimination via fine-grained understanding [25], and Fine-Grained OVOD [26] introduces a fine-grained, prompt-driven dataset and benchmark—these efforts largely remain at the level of optimizing prompting strategies and do not address the root cause: most existing works still use only category names as text inputs. Single words such as “chair,” “tomato,” and “car” can denote object categories, yet they provide only highly abstract descriptions of visual appearance. When two instances belong to the same category but differ in visual attributes, the semantic signal from the category name is nearly identical, leaving the model to rely primarily on visual features for discrimination. This issue is exacerbated for rare classes under long-tailed distributions, where the model lacks sufficient training samples and receives only coarse-grained words, making it difficult to establish accurate vision–semantic correspondences. In practice, the key cues for distinguishing near-duplicate targets lie in attribute-level semantics: descriptors such as “red,” “wooden,” and “rounded” map directly to observable visual patterns, whereas a single category name discards these discriminative signals.

Recent breakthroughs in Large Language Models (LLMs) provide a new direction for addressing this issue. Trained on massive text corpora, LLMs accumulate rich knowledge about object appearance, parts, and materials. Some studies have started to introduce LLMs into vision tasks; for example, LLMDet [27] uses semantic supervision generated by an LLM to train OVD detectors, and Lami-DETR [28] enhances detection capability through language-model instructions. However, these methods mainly rely on a single LLM to generate category descriptions or training data, leading to two key limitations. First, the output of a single model is constrained by its training corpus and inductive bias, resulting in systematic bias in the semantic coverage and expression precision of the generated attributes. Second, multiple LLMs can produce notably different descriptions for the same category; for instance, the Qwen family focuses on texture and structural features, the DeepSeek family tends to emphasize aesthetic and decorative aspects, and the Doubao family concentrates on core visual cues, making it difficult for a single source to guarantee the completeness and stability of the resulting attributes. Therefore, effectively integrating outputs from heterogeneous models to obtain a high-quality and diverse attribute set becomes a key challenge for building attribute-level semantics.

In this work, we do not view the main limitation of OVD as merely insufficient detector capacity. Instead, we argue that a fundamental bottleneck lies in the coarse semantic granularity of category-name-only text inputs. To address this issue, we propose a framework that constructs and injects appearance-oriented attribute priors for region-level discrimination. The novelty of our method lies not in introducing a new generic retrieval or fusion block in isolation, but in reframing OVD semantic enhancement as a collaborative attribute-selection problem and solving it with a cooperative multi-LLM game-theoretic mechanism under a compact CLIP-aligned semantic space. Concretely, the framework comprises two stages: a Concept Repository stage, which generates candidate attributes from heterogeneous LLMs and selects a compact, high-quality subset through cooperative optimization; and a feature enhancement stage, which retrieves relevant attributes and injects them into region-level visual features while preserving CLIP-aligned semantics. The enhanced features are then fed into a DETR-based detector, and a semantic consistency loss is introduced during training to maintain high-fidelity feature-category alignment. We intentionally focus on appearance semantics because attributes such as color, material, texture, shape, part structure, and size are directly grounded in local visual evidence and can therefore be injected into region-level features in a stable and interpretable manner. By contrast, semantic dimensions such as spatial relations, negation, and higher-order compositional reasoning rely more heavily on global context and multi-entity interactions, which are only partially captured by the current framework. We therefore position our method as an appearance-oriented semantic enhancement framework for OVD, rather than a general solution for all semantic reasoning dimensions. Experiments on LVIS, OV-COCO, RefCOCO, and OVDEval verify the effectiveness of the proposed framework.

The main contributions of our study are as follows:

• We revisit OVD from the perspective of semantic granularity, and show that category-name-only supervision is insufficient for attribute-level discrimination in long-tail and high-confusion settings.
• We propose a game-theoretic multi-LLM collaboration framework that constructs a compact attribute repository by jointly optimizing semantic quality, cross-model diversity, and redundancy through a cooperative max-min selection objective.
• We design an appearance-oriented semantic enhancement pipeline that injects retrieved attribute priors into region-level features while preserving CLIP-aligned semantics, yielding consistent gains on rare-category detection, attribute-aware grounding, and semantic-dimension evaluation.

2. Related Works

2.1. Open Vocabulary Object Detection

The core challenge of Open-Vocabulary Object Detection (OVD) is how to enable detectors to recognize categories that are unseen during training. Early zero-shot detection methods build a mapping from visual features to a semantic space using manually designed category attributes or word-vector representations, aiming to transfer knowledge through semantic relations. However, these approaches face two fundamental issues: the coverage and consistency of manually defined attributes are difficult to guarantee, and while word vectors can capture semantic relatedness between categories, they lack a direct correspondence with visual appearance.

The emergence of vision–language pre-trained models has shifted the technical direction. CLIP is trained on 400 million image–text pairs with contrastive learning and maps a visual encoder and a text encoder into a shared semantic space, allowing the model to assess semantic matching by computing the similarity between visual features and textual descriptions. How to transfer this image-level alignment capability to region-level detection then becomes a key issue. ViLD adopts a knowledge-distillation strategy, using classification logits from a CLIP teacher model to guide the detector’s region proposal network, enabling the detector to learn classification in the CLIP-aligned space. RegionCLIP approaches the problem from a data perspective by constructing large-scale pseudo-labeled region–text pairs and directly training a region encoder through region-level contrastive learning. Detic leverages rich category labels from image-classification data and converts image-level supervision into region-level training signals to expand the vocabulary. Although these methods differ in their technical routes, they generally share a semantic-input limitation: only category names are used as textual descriptions.

The limited semantic granularity of category names has become increasingly clear in recent studies. GLIP reformulates detection as a phrase-grounding task and increases semantic complexity by using full sentences as input, but the noun phrases in these sentences still remain at the category level. Grounding DINO introduces feature-enhancement modules and joint text–image encoding to improve cross-modal interaction at the architectural level, but it offers limited refinement of the textual semantics itself. DetCLIP makes full use of CLIP knowledge through teacher–student distillation and achieves performance gains, yet it does not address the semantic-granularity issue [29]. A common trait of these methods is that they focus text-prompt optimization on sentence structure and encoding strategies rather than on refining the semantic content. Experimental evidence reveals the limitation of this design choice: when the task requires attribute-level discrimination, such as distinguishing a “red wooden chair” from a “blue metal chair” in RefCOCO [30], the category name “chair” cannot provide discriminative cues about color or material, leading to a clear performance drop.

The root cause lies in the technical evolution of OVD, which has largely focused on how to leverage the alignment capability of vision–language pre-trained models while overlooking the design of semantic input content. Category names are highly abstract semantic symbols: their advantage is that they are simple and easy to obtain, but this level of abstraction is complemented by rich background knowledge in human cognition, whereas deep learning models find it difficult to infer fine-grained visual details from a single word. We address this issue from the perspective of semantic content by constructing, for each category, attribute-level semantic descriptions that include observable attributes such as color, material, and texture, thereby providing the detector with more discriminative textual inputs.

2.2. Vision Grounding and Multimodal Extensions

Recent vision grounding studies have demonstrated the importance of aligning flexible language expressions with specific visual regions. Remote Sensing Visual Grounding (RSVG) [31] extends visual grounding to remote sensing imagery, where the model is required to localize referred objects according to natural-language expressions. Compared with grounding in natural images, RSVG faces additional challenges such as large-scale variation, dense small objects, complex backgrounds, and limited annotations. Recent open-vocabulary RSVG studies further explore how foundation vision–language models can be used for zero-shot grounding in remote sensing scenarios.

In addition to domain-specific grounding, multimodal grounding has also attracted increasing attention. RGBT-Ground [32] introduces RGB-Thermal visual grounding for complex real-world scenarios, where aligned RGB and thermal infrared images are used to localize objects under challenging conditions such as illumination changes, nighttime scenes, weather variations, and long-distance observation. These works show that visual grounding is moving toward more open, robust, and multimodal region–language understanding.

Our work is related to these studies because it also relies on region–text alignment. However, the focus is different. Most visual grounding methods are given descriptive language expressions and aim to improve the localization or multimodal fusion architecture. In contrast, open-vocabulary object detection often uses only coarse category names as textual inputs, which lack fine-grained appearance cues. Our Concept Expander addresses this semantic limitation by injecting multi-source attribute priors, including color, material, texture, shape, and part structure, into the detection process. Therefore, our method is complementary to recent vision grounding works: instead of only improving the alignment module, it enriches the semantic content before alignment, enabling more attribute-aware open-vocabulary detection.

2.3. Attribute-Aware Recognition and Semantic Learning

Attribute-aware visual recognition focuses on distinguishing similar subcategories from subtle visual differences, which is closely related to the attribute-level discrimination required in OVD. Traditional methods can be grouped into three main lines: part-based modeling focuses on local discriminative features by locating key parts, such as a bird’s beak or wings; bilinear pooling captures higher-order statistical relations by computing outer products of feature channels; and attention mechanisms learn spatial or channel-wise weights to highlight discriminative regions. These methods are effective in closed-set classification tasks, but two limitations restrict their extension to open scenarios. First, they strongly depend on training data: part localization, bilinear weights, and attention distributions are all learned through supervised training on annotated data, and these mechanisms are difficult to adapt automatically to unseen categories. Second, they lack explicit semantic interpretation: although the models learn visual discriminative patterns, they struggle to associate them with clear attribute-level semantic concepts; as a result, when facing new categories, they cannot use semantic knowledge to guide feature extraction.

Attribute learning introduces interpretable intermediate representations into recognition tasks. This paradigm decomposes classification into two stages: the model first predicts observable attributes (e.g., “red,” “long beak,” and “striped”) and then infers the category based on the attribute combination. As semantic concepts, attributes have two key properties. Compositionality allows novel categories to be represented through different combinations of known attributes. Transferability enables attribute predictors learned on base classes to be applied to novel categories. Early works manually annotate an attribute vector for each category, and the model learns a mapping from visual features to attribute predictions. This approach shows zero-shot generalization on small-scale datasets; however, the cost of manual annotation increases linearly with the number of categories, making it difficult to scale to large-scale open settings.

Recent studies attempt to discover attributes through weakly supervised or unsupervised approaches. Some methods exploit the semantic relations between category names and attribute words in a word embedding space to automatically assign attribute labels to categories. Other works discover co-occurring discriminative patterns by clustering visual features and interpret the cluster centers as implicit attributes. These methods reduce annotation cost, but the discovered attributes often lack clear semantic meaning or have limited semantic coverage. Recent progress in Large Language Models (LLMs) provides a new path for attribute generation: LLMDet uses semantic supervision generated by LLM to train detectors, and Lami-DETR enhances detection capability through language-model instructions. However, existing methods mainly rely on a single LLM; the generated attributes are constrained by the model’s training distribution, and multiple LLMs show clear differences in semantic coverage and expression precision. In our study, we treat attributes as semantic signals for enhancing visual features and use a Concept Expander to enable dynamic retrieval and fusion of attributes.

The above studies reveal two core challenges faced by OVD. The first is the limitation of the semantic space: the embedding space of vision–language pre-trained models has a fixed dimensionality, and how to obtain more comprehensive attribute representations within this limited capacity becomes a key issue. Attributes generated by a single LLM are constrained by model bias and show insufficient semantic coverage; simply merging outputs from multiple LLMs introduces redundancy and reduces attribute quality; and how to use a collaborative selection mechanism to maximize semantic coverage while keeping the semantic space compact still lacks a systematic solution. The second challenge is the instability of attribute quality: different multiple LLMs produce notably different attribute descriptions for the same category, and selecting high-quality and complementary attribute combinations from heterogeneous model outputs remains an open problem, as existing methods lack effective evaluation and selection mechanisms. In this study, we use a game-theoretic framework to enable multi-LLM collaboration, which optimizes attribute selection within a limited semantic space and injects attribute-level semantics into visual features through a concept expander.

3. Methodology

Open-Vocabulary Object Detection (OVD) requires a model to take an image $\mathrm{I}\in {ℝ}^{\mathrm{H}\times \mathrm{W}\times 3}$ and an arbitrary text query $\mathrm{T}$ as input, and to output all objects in the image that are relevant to the query, together with their bounding boxes $\mathrm{b}=\left(\mathrm{x},\mathrm{y},\mathrm{w},\mathrm{h}\right)$ and corresponding confidence scores $\mathrm{s}\in \left[0,1\right]$. Unlike conventional detectors, OVD models are required to perform zero-shot detection on categories that are unseen during training.

The semantic granularity of category names is insufficient to support attribute-level discrimination. We propose a game-theoretic semantically enhanced detection method based on Multi-LLM Collaboration, and the overall framework is shown in Figure 1. The framework consists of two stages: offline concept generation and online detection. In the offline stage, attributes are generated in parallel by Heterogeneous Large Language Models, and a Concept Store is constructed via a Game-theoretic Framework. In the online stage, visual features are extracted from the input image; candidate regions are injected with attribute-level semantics through the Concept Expander and then fed into a Transformer-based detector to produce the final predictions.

The visual encoder uses CLIP ViT-B/16 as the backbone network. The input image is divided into patches and combined with positional encodings and is then passed through multiple Transformer encoder layers to extract global features $\mathrm{F}\in {ℝ}^{\mathrm{N}\times \mathrm{d}}$. The detection architecture is based on the end-to-end DETR framework and maintains a set of learnable object queries ${\left\{{\mathrm{q}}_{\mathrm{i}}\right\}}_{\mathrm{i}=1}^{\mathrm{M}}$, which interact with the encoded features through the decoder. The decoder adopts standard self-attention and cross-attention mechanisms, and the enhanced features produced by the Concept Expander are injected into the object queries through an additional cross-attention layer. The prediction head consists of a bounding-box regression branch and a classification branch: the former outputs normalized coordinates $\left({\mathrm{x}}_{\mathrm{c}},{\mathrm{y}}_{\mathrm{c}},\mathrm{w},\mathrm{h}\right)\in {[0,1]}^4$, while the latter projects the queries into the CLIP embedding space and computes similarity with text embeddings to obtain confidence scores. The detailed implementation of these standard components follows DETR and CLIP. In the following sections, we focus on the core innovations of our method.

3.1. Concept Repository

Attribute descriptions generated by a single LLM are constrained by the model’s training data and Inductive Bias. Different models show significant differences when describing the same category. For example, in our experiments, Qwen3-Max tends to generate details about texture and structure, DeepSeek-V3.2 focuses more on aesthetic and decorative descriptions, and Doubao-Seed-1.6 emphasizes core visual features. Using a single source makes it difficult to ensure that attributes are both comprehensive and consistently high-quality. This section presents a game-theoretic Multi-LLM Collaboration framework to build the Concept Store, with the overall process shown in Figure 2.

For a category set $\mathrm{C}=\left\{{\mathrm{c}}_1,{\mathrm{c}}_2,\dots ,{\mathrm{c}}_{\mathrm{N}}\right\}$, we select three Large Language Model (LLM) with different training corpora, $\left\{{\mathrm{LLM}}_1,{\mathrm{LLM}}_2,{\mathrm{LLM}}_3\right\}$, to generate attributes in parallel. Four task-oriented prompt templates guide the outputs toward identification cues, appearance descriptions, part structures, and scale information:

$$P=\left\{P_{id},P_{app},P_{part},P_{scale}\right\}$$

(1)

Specifically, ${\mathrm{P}}_{\mathrm{id}}$ is “How can you identify a(n) {class}?”, which guides the model to output discriminative features; ${\mathrm{P}}_{\mathrm{app}}$ is “Describe the visual appearance of a(n) {class}”, emphasizing surface features such as color and texture; ${\mathrm{P}}_{\mathrm{part}}$ is “What are the main parts of a(n) {class}?”, asking the model to list local components; and ${\mathrm{P}}_{\mathrm{scale}}$ is “Describe the size and proportions of a(n) {class}”, guiding the model to generate relative scale information. These prompt templates are intentionally designed to elicit visually grounded and region-observable attributes, because such attributes are most compatible with the downstream region-level retrieval and semantic injection process. This design further reflects our focus on appearance semantics rather than on relation-heavy or context-heavy reasoning cues.

For each model, the attribute set generated for category $\mathrm{c}$ is

$$A_i^c=LL{M}_i\left(c,P\right),\hspace{1em}i\in \left\{1,2,3\right\}$$

(2)

The generated responses are complete sentences containing many functional words. Using part-of-speech tagging, adjectives (JJ), nouns (NN/NNS), and gerunds (VBG) are retained, while prepositions, articles, and other functional words are filtered out:

$$\mathcal{R}\left(w,p\right)=\left\{\begin{array}{ll}1,\hfill & p\in \left\{NN,JJ,NNS,VBG\right\}\mathrm{and}w\notin S_{\mathrm{stop}}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(3)

The cleaned attribute words are mapped to 768-dimensional embeddings ${\mathrm{e}}_{\mathrm{a}}$ using a frozen CLIP text encoder and then L2-normalized. Freezing ensures encoding determinism and prevents embedding-space drift during training.

For each category $\mathrm{c}$, the three attribute sets ${\mathrm{A}}_1^{\mathrm{c}},{\mathrm{A}}_2^{\mathrm{c}}$, and ${\mathrm{A}}_3^{\mathrm{c}}$ generated by $\left\{{\mathrm{LLM}}_1,{\mathrm{LLM}}_2,{\mathrm{LLM}}_3\right\}$ exhibit different strengths in quality and diversity. Direct merging may introduce redundancy, while simple voting could discard high-quality attributes. We model attribute selection as a multi-agent cooperative game, where each model is a participant aiming to find an attribute subset that jointly maximizes the utility of all participants. Each participants utility function considers three dimensions:

$$U_i\left(S\right)=\alpha \cdot Q_i\left(S\right)+\beta \cdot D_i\left(S\right)-\gamma \cdot R_i\left(S\right)$$

(4)

To explicitly clarify, the utility components are quantitatively measured based on the cross-modal cosine similarity of the attributes within the pre-trained CLIP text embedding space, rather than relying on an external pre-trained reward model. This ensures the selected attributes are naturally aligned with the semantic space used by the downstream visual encoder.

The semantic quality term measures whether the selected attributes are semantically coherent with the target category and with the attribute set contributed by the corresponding LLM. The diversity term measures the semantic complementarity between attributes generated by different LLMs, encouraging the final subset to cover different visual aspects such as color, material, texture, shape, and part structure. The redundancy term penalizes attributes that are highly similar to already selected attributes from the same source. Therefore, the attribute utility is a CLIP-embedding-based semantic utility rather than a reward-model score.

During the evolution process, the marginal gain of a candidate attribute a with respect to the current selected set S is defined as follows: Here, $\mathrm{S}$ denotes the candidate final attribute set. The quality term ${\mathrm{Q}}_{\mathrm{i}}\left(\mathrm{S}\right)$ measures the average cohesion of the attributes contributed by ${\mathrm{LLM}}_{\mathrm{i}}$ within $\mathrm{S}$:

$$Q_i\left(S\right)={\displaystyle \frac{1}{{\left|S\cap A_i^c\right|}^2}}{\displaystyle \sum }_{a,a^{\prime }\in S\cap A_i^c}{e}_a^{\top }{e}_{a^{\prime }}$$

(5)

The diversity term ${\mathrm{D}}_{\mathrm{i}}\left(\mathrm{S}\right)$ quantifies this model’s contribution to the semantic coverage of $\mathrm{S}$ by computing the semantic dissimilarity between the selected attributes and the attributes from other models:

$$D_i\left(S\right)={\displaystyle \frac{1}{\left|S\cap A_i^c\right|}}{\displaystyle \sum }_{a\in S\cap A_i^c}\left(1-\underset{a^{\prime }\in S\backslash A_i^c}{\max }{e}_a^{\top }{e}_{a^{\prime }}\right)$$

(6)

The redundancy term ${\mathrm{R}}_{\mathrm{i}}\left(\mathrm{S}\right)$ penalizes internal repetition among the attributes contributed by this model:

$$R_i\left(S\right)={\displaystyle \frac{1}{\left|S\cap A_i^c\right|}}{\displaystyle \sum }_{a\in S\cap A_i^c}\underset{a^{\prime }\in S\cap A_i^c,a^{\prime }\ne a}{\max }{e}_a^{\top }{e}_{a^{\prime }}$$

(7)

The cooperative Nash equilibrium corresponds to the attribute set that maximizes the minimum utility across all models, thereby ensuring fairness:

$$S_e^{*}=\arg \underset{S\subseteq {\cup }_i{A}_i^c}{\max }\underset{i\in \left\{1,2,3\right\}}{\min }{U}_i\left(S\right)$$

(8)

This optimization problem is approximated via an iterative greedy max–min algorithm. Starting from ${\mathrm{S}}^{\left(0\right)}=⌀$, at each step we add the attribute that yields the largest gain in the minimum utility:

$$a^{*}=\arg \underset{a\in {\cup }_i{A}_i^c\backslash S^{\left(t\right)}}{\max }\underset{i}{\min }\left(U_i\left(S^{\left(t\right)}\cup \left\{a\right\}\right)-U_i\left(S^{\left(t\right)}\right)\right)$$

(9)

Exhaustive search over all possible attribute subsets is computationally intractable. Therefore, for each category, we approximate the cooperative Nash solution with an iterative greedy max–min procedure. Let $\mathrm{C}$ denote the candidate pool obtained by merging the filtered attributes from all LLMs, and let ${\mathrm{S}}_{\mathrm{t}}$ be the selected subset at iteration $\mathrm{t}$. For each candidate attribute $\mathrm{a}\in \mathrm{C}\backslash {\mathrm{S}}_{\mathrm{t}}$, we compute its marginal contribution to the minimum agent utility as

$$\mathrm{g}\left(\mathrm{a};{\mathrm{S}}_{\mathrm{t}}\right)=\underset{\mathrm{i}}{\min }{\mathrm{U}}_{\mathrm{i}}\left({\mathrm{S}}_{\mathrm{t}}\cup \left\{\mathrm{a}\right\}\right)-\underset{\mathrm{i}}{\min }{\mathrm{U}}_{\mathrm{i}}\left({\mathrm{S}}_{\mathrm{t}}\right)$$

(10)

We then select

$${\mathrm{a}}^{*}=\arg \underset{\mathrm{a}\in \mathrm{C}\backslash {\mathrm{S}}_{\mathrm{t}}}{\max }\mathrm{g}\left(\mathrm{a};{\mathrm{S}}_{\mathrm{t}}\right),$$

(11)

and update ${\mathrm{S}}_{\mathrm{t}+1}={\mathrm{S}}_{\mathrm{t}}\cup \left\{{\mathrm{a}}^{*}\right\}$ if $\mathrm{g}\left({\mathrm{a}}^{*};{\mathrm{S}}_{\mathrm{t}}\right)>\mathsf{\epsilon }$. Otherwise, the algorithm terminates. In practice, the iteration stops when either the selected subset reaches a preset maximum size ${\mathrm{K}}_{\max }$ or the best marginal gain falls below the threshold $\mathsf{\epsilon }$. In our implementation, $\mathsf{\epsilon }=0.01$. Since each accepted step monotonically increases the minimum utility and the candidate pool is finite, the procedure terminates in a finite number of iterations. We therefore refer to the resulting subset as an approximate cooperative Nash solution used for concept repository construction.

The process is repeated until $\left|{\mathrm{S}}_{\mathrm{c}}^{*}\right|$ reaches a preset size or the utility gain falls below a threshold. The above game-theoretic optimization procedure is applied to all categories, producing an optimal attribute set ${\mathrm{S}}_{\mathrm{c}}^{*}$ for each category $\mathrm{c}$. After removing duplicates using a similarity threshold of 0.85, the optimal attribute sets across all categories are merged into the Concept Store ${\mathrm{C}}_{\mathrm{repo}}=\bigcup _{\mathrm{c}}{\mathrm{S}}_{\mathrm{c}}^{*}$, which contains around 5000 attribute concepts.

The attributes are converted into embedding vectors through the CLIP text encoder and stored in the Concept Store. Specifically, each attribute $\mathrm{a}\in {\mathrm{C}}_{\mathrm{repo}}$ corresponds to a 768-dimensional L2-normalized CLIP embedding ${\mathrm{e}}_{\mathrm{a}}\in {ℝ}^{768}$, referred to as an attribute embedding or concept. For convenient retrieval and matching, the $\mathrm{K}=\left|{\mathrm{C}}_{\mathrm{repo}}\right|$ attribute embeddings in the store are indexed sequentially as ${\left\{{\mathrm{y}}_{\mathrm{j}}\right\}}_{\mathrm{j}=1}^{\mathrm{K}}$, where ${\mathrm{y}}_{\mathrm{j}}={\mathrm{e}}_{{\mathrm{a}}_{\mathrm{j}}}$ represents the embedding of the $\mathrm{j}$-th attribute.

Unlike naive aggregation or fixed-weight fusion, the cooperative game framework reaches a joint equilibrium—specifically a cooperative Nash equilibrium—where no single model’s distribution bias can dominate the output. Each LLM acts as a cooperative agent: the max–min objective ensures that even the least-represented model retains meaningful coverage in the final attribute set, simultaneously optimizing semantic quality and cross-source diversity within a compact semantic space.

3.2. Concept Expander

The Concept Expander injects attribute semantics from the Concept Store into region-level visual features, as shown in Figure 3. Given region features ${\mathrm{x}}_{\mathrm{i}}\in {ℝ}^{\mathrm{d}}$ extracted via RoIAlign [33] and a store containing $\mathrm{K}$ attribute embeddings ${\left\{{\mathrm{y}}_{\mathrm{j}}\right\}}_{\mathrm{j}=1}^{\mathrm{K}}$, the goal is to dynamically retrieve the concepts most relevant to the current region and integrate them into the visual representation.

We intentionally adopt a lightweight retrieval design rather than introducing a heavily customized learnable retrieval architecture. This choice is deliberate for two reasons. First, the goal of this work is to isolate and validate the contribution of the collaboratively constructed attribute priors, rather than to entangle it with additional retrieval engineering. Second, cosine-similarity retrieval in the frozen CLIP-aligned semantic space preserves the pretrained vision-language geometry, which is important for stable transfer across categories and datasets. Therefore, the retrieval module is kept simple and interpretable, while the main methodological novelty is placed on multi-LLM attribute construction and game-theoretic selection.

To compute feature similarity in the pre-trained CLIP semantic space, the region features are first mapped to the CLIP text space using an alignment projection layer ${\mathrm{W}}_{\mathrm{align}}\in {ℝ}^{768\times \mathrm{d}}$.

$$x_i^{\mathrm{clip}}=W_{\mathrm{align}}{x}_i\in {ℝ}^{768}$$

(12)

For each projected region feature, we compute the cosine similarity with all attribute embeddings:

$${\mathrm{score}}_{i,j}={\displaystyle \frac{{\left(x_i^{\mathrm{clip}}\right)}^{\top }{y}_j}{\parallel x_i^{\mathrm{clip}}\parallel \parallel y_j\parallel }},\hspace{1em}j=1,\dots ,K$$

(13)

The resulting similarity matrix score $\in {ℝ}^{\mathrm{M}\times \mathrm{K}}$ describes the semantic association strength between each region and each concept. We use a Top-$\mathrm{k}$ operation to select the $\mathrm{k}$ concepts with the highest similarity, filtering out noise from irrelevant concepts. The selected concepts are then normalized via softmax to compute a weight distribution:

$$w_{i,j}={\displaystyle \frac{\exp \left({\mathrm{score}}_{i,j}/\tau \right)}{{{\displaystyle \sum }}_{j^{\prime }=1}^k\exp \left({\mathrm{score}}_{i,j^{\prime }}/\tau \right)}}$$

(14)

where $\mathsf{\tau }$ is a temperature parameter controlling the concentration of the weight distribution. After obtaining the normalised weights, a semantic prototype vector is generated by weighted aggregation:

$$s_i={\displaystyle {\displaystyle \sum }_{j=1}^k}{w}_{i,j}\cdot y_j\in {ℝ}^{768}$$

(15)

This semantic prototype aggregates the attributes most relevant to the current region. Through a residual connection, the semantic prototype is injected into the aligned feature, enhancing the feature in the CLIP space:

$$h_i=x_i^{\mathrm{clip}}+\alpha \cdot s_i\in {ℝ}^{768}$$

(16)

Here, $\mathsf{\alpha }$ controls the strength of semantic injection. The residual structure ensures that, even if semantic retrieval is unstable at the early training stage, gradients can still propagate along the visual path. Finally, the enhanced feature is mapped back to the detector feature space through a projection layer ${\mathrm{W}}_{\mathrm{proj}}\in {ℝ}^{\mathrm{d}\times 768}$:

$$x_i^{\exp }=W_{proj}{h}_i\in {ℝ}^d$$

(17)

The enhanced features $\left\{{\mathrm{x}}_{\mathrm{i}}^{\exp }\right\}$ are fed into the Transformer decoder, where cross-attention with object queries is used to perform classification and localization. In this sense, the Concept Expander should be viewed as a semantic injection interface built on a simple retrieval pipeline, rather than as a standalone contribution in retrieval architecture design. Because the retrieved concepts are matched to RoI-level features, the module is naturally best suited to appearance-grounded attributes that have direct local visual evidence, rather than to semantic relations that require broader scene-level reasoning.

3.3. Training

The training objective is a multi-task joint optimisation, with the total loss composed of classification, localization, and semantic alignment terms:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{cls}}+{\lambda }_{\mathrm{box}}{\mathcal{L}}_{\mathrm{box}}+{\lambda }_{\mathrm{align}}{\mathcal{L}}_{\mathrm{align}}$$

(18)

The classification loss uses Focal Loss [34] to alleviate class imbalance in long-tailed distributions. The localization loss combines L1 and GIoU [35] to ensure coordinate accuracy and scale invariance, following standard detection frameworks. The semantic consistency loss constrains the enhanced features to remain consistent with the true class. For foreground predictions $\mathrm{i}\in {\mathcal{M}}_{\mathrm{fg}}$ that are correctly matched, the concept center of the true class ${\mathrm{c}}_{\mathrm{i}}$ is obtained by averaging the embeddings of its attributes:

$$t_{c_i}={\displaystyle \frac{1}{\left|S_{c_i}^{*}\right|}}{\displaystyle \sum }_{a\in S_{c_i}^{*}}{e}_a$$

(19)

The alignment loss is implemented by maximising the cosine similarity between the enhanced features and the concept center:

$${\mathcal{L}}_{\mathrm{align}}=-{\displaystyle \frac{1}{\left|{\mathcal{M}}_{fg}\right|}}{\displaystyle \sum }_{i\in {\mathcal{M}}_{fg}}{\displaystyle \frac{x_i^{exp}\cdot t_{c_i}}{\parallel x_i^{exp}\parallel \parallel t_{c_i}\parallel }}$$

(20)

This constraint ensures that the transformation learned by the semantic projection layer does not deviate from the category semantic space, allowing attributes to enhance rather than replace visual discriminative power.

Training is performed on the 48 base classes of OV-COCO, and the CLIP encoder weights are frozen to preserve pre-trained vision–language alignment knowledge. Training is performed on the 48 base classes of OV-COCO, while evaluation on LVIS rare classes is used to measure transfer to a broader cross-domain long-tail vocabulary. The two projection layers in the Concept Expander are trained from random initialization: $W_{\mathrm{align}}$ learns the alignment mapping from detection features to the CLIP space using a smaller learning rate to ensure training stability, whereas $W_{\mathrm{proj}}$ learns the projection from enhanced features to the detection space using a standard learning rate.

4. Experiments

4.1. Datasets and Evaluation Indicators

The experiments are conducted on four benchmark datasets, each with a different evaluation focus. The LVIS dataset contains 1203 categories, divided into 866 base classes and 337 novel classes; the metric APr specifically measures performance on rare classes, evaluating the model’s generalization ability to fine-grained categories. The OV-COCO dataset is split into 48 base classes and 17 novel classes, with APnovel reflecting open-vocabulary generalization. The RefCOCO series requires the model to localize targets based on natural-language descriptions that include attribute-level details, with Accuracy@0.5IoU as the evaluation metric, assessing the model’s understanding of visual attributes such as appearance priors and state features. The OVDEval dataset evaluates along five semantic dimensions—color, material, location, relations, and negation—calculating AP50 for each dimension individually, providing a comprehensive assessment of the model’s attribute-aware capability. Unless otherwise stated, the detector is trained on the 48 base classes of OV-COCO, and evaluation on LVIS rare classes is used to assess cross-domain generalization from a limited base vocabulary to a broader long-tail category space.

4.2. Implementation Details

The proposed method is built upon the CORA detection framework, retaining its base Transformer encoder–decoder architecture and introducing the Concept Expander module. CORA uses ResNet-50 as the visual encoder, which we replace with a CLIP ViT-B/16 pre-trained on the LAION-400M dataset [36] to achieve stronger visual–language alignment. The original 6-layer encoder and 6-layer decoder architecture and 300 object queries of CORA are preserved. In the decoder, a cross-attention layer between object queries and the enhanced features is added to inject attribute-level semantics.

The attribute repository is pre-constructed before training. We select three heterogeneous large language models—Qwen3-Max, DeepSeek-V3.2, and Doubao-Seed-1.6—to generate 5–10 visual attributes for the 65 OV-COCO classes and 1203 LVIS classes, respectively. Each model generates attributes in parallel using four prompt templates. After part-of-speech filtering, stopword removal, and CLIP encoding, attributes are selected via a game-theory collaboration framework. The utility function balance coefficients are set as, and the stopping threshold for the iterative greedy algorithm is 0.01. After filtering and removing duplicates with a similarity threshold of 0.85, about 5000 concepts remain, covering the semantic space of both training and evaluation classes.

We selected Qwen3-Max, DeepSeek-V3.2, and Doubao-Seed-1.6 as three representative heterogeneous LLMs rather than claiming them to be the universally best-performing models. Our selection criterion was diversity of attribute-generation behavior: in pilot observations, Qwen3-Max tended to produce texture- and structure-oriented descriptions, DeepSeek-V3.2 emphasized aesthetic and decorative properties, and Doubao-Seed-1.6 generated concise core visual cues. This complementarity makes them suitable agents for the proposed collaborative selection framework. The framework itself is model-agnostic and can be extended to other heterogeneous LLM combinations.

The Concept Expander contains two learnable linear transformations: an alignment projection layer ${\mathrm{W}}_{\mathrm{aligm}}\in {ℝ}^{768\times \mathrm{d}}$ that maps region features into the CLIP space for similarity computation, and a detector projection layer ${\mathrm{W}}_{\mathrm{proj}}\in {ℝ}^{\mathrm{d}\times 768}$ that projects the enhanced features back to the detector dimension. For attribute retrieval, Top-$\mathrm{k}$ is set to $\mathrm{k}=20$, the temperature parameter $\mathsf{\tau }=0.07$, and the fusion coefficient $\mathsf{\alpha }=0.5$. Training is performed on the 48 base classes of OV-COCO using the AdamW optimizer, with the learning rate cosine-annealed from $1\times {10}^{-4}$ to $1\times {10}^{-6}$ over 100 epochs. During multi-scale training, the short side of input images is randomly sampled between 800 and 1333 pixels. The initial loss weights are set as ${\mathsf{\lambda }}_{\mathrm{box}}=5.0$ and ${\mathsf{\lambda }}_{\mathrm{align}}=1.0$. Experiments are conducted on a system with 2$\times$ NVIDIA A100 40GB GPUs, Ubuntu 20.04, CUDA 11.7, and PyTorch 2.0.

4.3. Comparative Experiment

The LVIS dataset, with its long-tailed distribution of 1203 categories, serves as an important benchmark for evaluating open-vocabulary detectors. In our setting, the detector is trained on the 48 base classes of OV-COCO and then evaluated on LVIS rare classes.

Table 1. Zero-Shot Detection Performance on LVIS.

Method	Backbone	Backbone Size	APr	AP
YOLO-World-S	YOLOv8-S	77 M	16.4	24.2
OADP	RN50	26 M	21.9	28.7
CoDet	RN50	26 M	23.4	30.7
LBP	RN50	26 M	24.1	29.9
CAKE	RN50	26 M	25.0	34.9
RegionCLIP	RN50x4	87 M	22.0	32.3
CORA	RN50x4	87 M	22.2	-
VLDet	Swin-B	88 M	26.3	38.1
CoDet	Swin-B	88 M	29.4	39.2
CLIPSelf	ViT-B/16	86 M	25.3	-
RO-ViT	ViT-B/16	86 M	28.4	31.9
CFM-ViT	ViT-B/16	86 M	29.6	33.8
OV-DQUO	ViT-B/16	86 M	29.7	-
DeCLIP	ViT-B/16	86 M	31.0	27.7
ours	ViT-B/16	86 M	28.5	32.3

shows a comparison of zero-shot detection performance. Our method achieves 28.5 APr on rare classes, improving by 6.3 points over the CORA baseline of 22.2, corresponding to a relative increase of 28.3%, and reaches an overall AP of 32.3. This improvement demonstrates that the Concept Expander compensates for the limited training data in low-resource categories by introducing attribute-level semantic priors. For models with a ViT-B/16 backbone, our method’s APr surpasses RO-ViT’s 28.4 [37] and is close to CFM-ViT’s 29.6 [38] and OV-DQUO’s 29.7 [39], though slightly lower than DeCLIP’s 31.0 [40]. It is noteworthy that DeCLIP employs a complex two-stage knowledge distillation training, while our method achieves competitive results through a single-stage end-to-end training under a simpler paradigm. Across different backbones, within the ResNet-50 series, CAKE [41] achieves 25.0 APr, and within the Swin-B series, CoDet [42] reaches 29.4 APr. Our method achieves 28.5 APr and an overall AP of 32.3 under the ViT-B/16 backbone, surpassing RO-ViT’s 31.9 AP, demonstrating a good balance between rare-class performance and overall detection performance.

The OV-COCO dataset, divided into 48 base classes and 17 novel classes, allows for a more focused evaluation of zero-shot generalization.

Table 2. Open-Vocabulary Detection Performance on OV-COCO.

Method	Backbone	APnovel	APbase	AP
CoDet	RN50	30.6	52.3	46.6
SAS-Det	RN50	37.4	58.5	53.0
LBP	RN50	37.8	58.7	53.2
CAKE	RN50	41.8	60.6	55.7
RegionCLIP	RN50x4	39.3	61.6	55.7
CORA	RN50x4	41.7	44.5	43.8
OV-DQUO	RN50x4	45.6	-	-
F-ViT	ViT-B/16	37.6	54.9	50.4
DeCLIP	ViT-B/16	46.1	56.3	53.6
ours	ViT-B/16	44.5	61.2	56.8

presents the open-vocabulary detection performance. Our method achieves 44.5 APnovel on novel classes, improving by 2.8 points over the CORA baseline of 41.7, demonstrating that the Concept Expander helps transfer knowledge learned from base classes to unseen categories through attribute-level semantic enhancement. The overall AP reaches 56.8, an increase of 13.0 points over the baseline of 43.8, achieving a good balance between base and novel classes. For base classes, our method achieves 61.2 AP, surpassing CAKE’s 60.6 and approaching RegionCLIP’s 61.6. Among ResNet-50 backbone methods, CAKE reaches 41.8 AP on novel classes. Our method with a ViT-B/16 backbone achieves 44.5 AP on novel classes, demonstrating the zero-shot generalization advantage brought by CLIP pre-training. Although the AP on novel classes is slightly lower than DeCLIP’s 46.1 and OV-DQUO’s 45.6, our method shows better balance between base class and overall AP.

The RefCOCO series requires models to precisely localize targets based on natural language descriptions that include attribute-level details, making it a key benchmark for evaluating attribute-aware understanding.

Table 3. Attribute-Aware Localisation Performance on RefCOCO (Acc@0.5).

Method	RefCOCO			RefCOCO+			RefCOCOg
	Val	TestA	TestB	Val	TestA	TestB	Val	Test
GLIP	50.4	54.3	43.8	49.5	52.8	44.6	66.1	66.9
Grounding DINO	50.8	57.4	45.0	51.6	57.3	46.4	60.4	59.7
ours	54.8	60.8	48.8	52.3	57.8	46.8	68.2	68.6

shows the localization accuracy comparison. On the RefCOCO validation set, our method achieves 54.8, improving by 4.0 points over Grounding DINO’s 50.8 and 4.4 points over GLIP’s 50.4. On the testA and testB subsets, it reaches 60.8 and 48.8, respectively, improving by 3.4 and 3.8 points over Grounding DINO’s 57.4 and 45.0. For RefCOCO+, the val, testA, and testB subsets achieve 52.3, 57.8, and 46.8, while RefCOCOg reaches 68.2 on val and 68.6 on test. These improvements demonstrate that the Concept Expander enables the model to learn correspondences between semantic descriptions in the concept repository—such as color and material—and visual appearance, allowing for more accurate focus on relevant visual regions when processing descriptive language.

To further probe the method’s attribute sensitivity, we evaluate on OVDEval across five semantic dimensions: color, material, position, relationship, and negation (

Table 4. Semantic-Dimension Performance on OVDEval (AP50).

Method	Color	Material	Position	Relationship	Negation
GLIP	3.7	7.4	30.9	10.0	29.3
Grounding DINO	9.4	9.0	67.5	10.7	52.5
Detic	3.1	2.5	34.0	8.2	28.3
ours	12.3	11.2	36.8	12.4	33.6

). The most pronounced gains appear on color (+2.9 over Grounding DINO, reaching 12.3) and material (+2.2, reaching 11.2), consistent with the attribute repository’s color- and texture-rich vocabulary directly strengthening these perceptual channels. Improvements on the position and relationship dimensions are moderate, which aligns with the method’s design scope—the Concept Expander enriches instance-level appearance priors rather than spatial relational reasoning. The negation dimension (33.6) outperforms GLIP and Detic but remains low overall, reflecting a field-wide limitation rather than a method-specific one. Taken together, OVDEval confirms the method targets precisely the dimensions it is designed to improve.

While quantitative metrics can reflect overall performance improvements, they are insufficient to intuitively demonstrate the specific performance differences of the model in attribute-level discrimination scenarios. To more clearly present the semantic enhancement effect introduced by the concept expander, we selected typical samples from the LVIS rare class and the OV-COCO novel class for visual comparison. Figure 4 shows a comparison of the detection results of our proposed method (left) and the baseline method (right) across multiple fine-grained categories. It can be observed that our proposed method exhibits a significant advantage in attribute-level discrimination, more accurately locating and identifying targets with similar appearances but different attributes. This is attributed to the concept expander providing the model with richer discriminative cues through the attribute-level semantic information injected by the concept repository.

Figure 4 shows the qualitative comparison results on the LVIS dataset. The visualization results demonstrate that our method has a significant advantage in rare category recognition, successfully detecting long-tailed objects such as “chessboard,” “handsaw,” and “fish tank,” while baseline methods struggle to effectively identify such rare categories in these scenes. However, our method misses some small targets, occlusions, or semantically similar categories, reflecting that current methods still have room for improvement in balancing fine-grained category coverage with small target perception capabilities.

Despite the consistent gains brought by the proposed framework, several limitations remain. First, the method is less reliable on small or low-resolution objects, because the local visual evidence is too weak to support stable retrieval of fine-grained appearance attributes. Second, performance degrades in heavily occluded or cluttered scenes, where only partial object regions are visible and the retrieved concepts may be matched to incomplete or noisy features. Third, failures also occur for semantically adjacent categories or cases dominated by non-appearance cues, where distinctions depend more on global context, object relations, or scene-level reasoning than on local appearance attributes alone. These observations are consistent with the design scope of our method: the Concept Expander is strongest when discriminative cues are appearance-grounded, but is less effective when recognition depends on broader contextual or relational semantics.

4.4. Ablation Analysis

Table 5. Ablation Study of Core Modules.

Configuration	LVIS APr	LVIS AP	RefCOCO
CORA baseline (ResNet-50)	22.2	-	-
+CLIP ViT-B/16	24.3	28.5	51.3
+Single LLM Concept Repository	26.8	31.2	53.6
+Multi-LLM Simple Merge	27.2	31.5	53.9
+Game-Theoretic Collaboration Framework	27.8	31.9	54.5
+Semantic Consistency Loss (Full Model)	28.5	32.3	54.8

presents a progressive ablation study of the core modules. The CORA baseline achieves 22.2 APR on rare classes in the LVIS dataset. Replacing the visual encoder with CLIP ViT-B/16 leads to a performance jump—APr increases to 24.3 and overall AP reaches 28.5—mainly due to CLIP’s strong visual–language alignment learned from large-scale image–text data. Introducing a single-LLM attribute repository further improves APr to 26.8, overall AP to 31.2, and RefCOCO to 53.6, demonstrating the effectiveness of attribute-level semantic injection. Simply merging multiple LLMs raises APr to 27.2, overall AP to 31.5, and RefCOCO to 53.9, showing that multi-source attributes bring improvements but also introduce redundancy. With the game-theoretic collaborative framework, APr significantly increases to 27.8, overall AP to 31.9, and RefCOCO to 54.5, as the cooperative Nash equilibrium mechanism balances quality and diversity through collaborative optimization. Finally, adding the semantic consistency loss leads to 28.5 APr, 32.3 overall AP, and 54.8 on RefCOCO, ensuring alignment between enhanced features and the true class semantics.

Table 6. Ablation Study for Separating Multi-LLM Diversity and Game-Theoretic Selection.

LLM Setting	LLM Configuration	Concept Repository Size	Quality ScoreQ	Diversity D	RefCOCO	LVIS APr
Single LLM	Qwen3-Max only	4200	72.3	2.18	53.6	27.1
	DeepSeek-V3.2 only	3900	69.8	2.31	53.2	26.8
	Doubao-Seed-1.6 only	4100	71.5	2.24	53.4	26.9
Multiple LLMs	Simple merge	6200	68.5	2.89	53.8	27.3
	Weighted fusion	5800	74.2	2.67	54.0	27.5
	Game-theoretic (Nash equilibrium)	5000	76.8	2.95	54.8	28.5

compares the effects of different LLM configurations on the quality of the concept repository. Attributes generated by a single LLM score lower on both quality Q and diversity D compared to multi-model collaborative approaches. Using Qwen3-Max alone achieves a quality score of 72.3 but a diversity of only 2.18, indicating that the generated attributes are semantically focused but have limited coverage. DeepSeek-V3.2 shows slightly higher diversity (2.31) but lower quality (69.8). Simply merging the three models combines all attributes, expanding the repository to 6200 entries. Although diversity increases to 2.89, quality drops to 68.5 due to noise from low-quality and duplicate attributes, resulting in RefCOCO at 53.8 and LVIS APr at 27.3. Weighted fusion with fixed weights reduces the repository to 5800 attributes, restoring quality to 74.2 but reducing diversity to 2.67, achieving 54.0 on RefCOCO and 27.5 APr on LVIS. The game-theoretic collaborative framework solves a Nash equilibrium, maintaining 5000 high-quality attributes while raising quality to 76.8 and diversity to 2.95. Combined with the full system, it reaches 54.8 on RefCOCO and 28.5 APr on LVIS, improving over a single LLM by 1.2 and 1.4 points and over simple merging by 1.0 and 1.2 points, validating the framework’s effectiveness in jointly optimizing quality and diversity.

To more intuitively illustrate the advantages of the game-theoretic collaborative framework in attribute generation, Figure 5 presents a comparison for the category “black leather sofa”. The figure shows the attribute words generated by three individual LLMs and by the collaborative framework.

As shown in Figure 5, Qwen3-Max tends to generate attributes describing texture and structural details (smooth, glossy, firm, flexible, tufted, stitching, cushions, backrest, armrests, frame, legs), offering comprehensive semantics but somewhat verbose. DeepSeek-V3.2 focuses on aesthetic and decorative descriptions (sleek, modern, patina, nailhead trim, walnut, brushed nickel, sophisticated, elegant), emphasizing design and refinement. Doubao-Seed-1.6 highlights core visual discriminative features (black, leather, grain, matte, minimalist, refined, taut, stitching), which is concise but may miss some details. The game-theoretic collaborative framework selects the optimal combination from the outputs of the three LLMs via a Nash equilibrium, preserving high-quality structural attributes (backrest, armrests, cushions, frame) while covering material (leather), color (black), texture (smooth, tufted), style (modern, minimalist), and decorative details (stitching, grain). This achieves a balanced optimization of both semantic coverage and attribute quality.

Table 7. Ablation Study on Concept Repository Size.

Concept Repository Size	RefCOCO	LVIS APr	Retrieval Time (ms)
1000	52.5	26.8	0.8
5000	54.8	28.5	2.1
10,000	54.6	28.5	4.3

analyzes the impact of the concept repository size on performance. When the number of attributes is 1000, RefCOCO accuracy is 52.5 and LVIS APr is 26.8, indicating that the semantic space is too sparse for the model to retrieve sufficiently relevant attribute descriptions. Expanding to 5000 attributes achieves the best balance, with RefCOCO at 54.8 and APr at 28.5, and a retrieval time of 2.1 ms, which is acceptable. Further increasing to 10,000 attributes shows almost no performance gain; RefCOCO slightly drops to 54.6, while retrieval time doubles to 4.3 ms, indicating that 5000 attributes sufficiently cover the semantic space and further expansion introduces redundancy.

Table 8. Ablation Study on Similarity Deduplication Threshold.

Similarity Threshold	RefCOCO	LVIS APr	Repository Size
0.75	54.2	28.0	3200
0.85	54.8	28.5	5000
0.95	53.8	27.8	7800

shows the effect of the similarity deduplication threshold. A threshold of 0.75 is too strict, retaining only 3200 attributes, resulting in RefCOCO at 54.2 and APr at 28.0, with slight performance degradation. A threshold of 0.85 achieves a balance between preserving semantic diversity and removing redundancy, yielding RefCOCO 54.8 and APr 28.5. A threshold of 0.95 is too loose; keeping 7800 attributes reduces RefCOCO to 53.8 and APr to 27.8, as near-synonyms interfere with precise retrieval.

Figure 6 shows the effect of different temperature coefficients on performance. When τ = 0.01, RefCOCO accuracy is 52.8 and LVIS APr is 27.2, as the weight distribution concentrates too much on a few highly similar attributes. Increasing τ to 0.04 improves performance to 54.1 and 28.0. At τ = 0.07, the best performance is achieved with RefCOCO 54.8 and APr 28.5. Further increasing τ to 0.10 reduces RefCOCO to 54.3 and APr to 28.1, and at τ = 0.13, performance drops to 53.7 and 27.6, because the weight distribution becomes too smooth and irrelevant attributes receive excessive weight. These results indicate that τ = 0.07 achieves the optimal balance between concentration and dispersion.

Figure 7 shows the effect of different Top-k values on performance. When k = 5, RefCOCO accuracy is 52.1 and LVIS APr is 26.8, with a fusion time of 0.6 ms, as too few attributes are retrieved. Increasing k to 10 improves performance to 53.6 and 27.5, with a fusion time of 1.1 ms. At k = 20, the best performance is achieved, with a RefCOCO accuracy of 54.8 and an APr accuracy of 28.5, and a fusion time of 2.1 ms. Further increasing k to 35 slightly reduces performance to 54.2 and 28.2, with a fusion time of 3.2 ms. At k = 50, performance drops to 53.7 and 27.9, and fusion time rises to 5.3 ms. These results indicate that k = 20 provides the best balance between performance and efficiency. Smaller k values retrieve too few attributes, while larger k values introduce less relevant attributes and increase computational cost. Experiments show that the method is robust to the choice of k, with stable performance in the 10–35 range.

Figure 8 illustrates the impact of the semantic fusion coefficient α, which controls the strength of attribute information injection and directly affects the balance between semantic and visual components in the enhanced features. When α = 0.0, no semantic information is injected, reducing the model to a purely visual baseline, with a RefCOCO accuracy of 51.3 and an LVIS APr of 25.8. At α = 0.3, semantic injection is weak, yielding a RefCOCO accuracy of 53.5 and an APr of 27.2. α = 0.5 achieves the optimal balance, with RefCOCO 54.8, APr 28.5, and OVDEval Color 12.3. At α = 0.8, semantic injection is too strong, slightly lowering performance to RefCOCO 54.2 and APr 28.0. When α = 1.0, semantic and visual components are equally weighted, and RefCOCO drops to 53.8, indicating that overemphasizing semantics can weaken the dominance of visual appearance.

Table 9. Inference Efficiency Comparison.

Method	GFLOPs	Reasoning Time (ms)	FPS	LVIS APr
CORA (baseline)	186	38.4	26.0	22.2
YOLO-World-L	168	19.2	52.1	27.6
ours	245	58.2	17.2	28.5

presents a comparison of inference efficiency. Our method requires 245 GFLOPs, a 31.7% increase over the CORA baseline’s 186 GFLOPs, with an inference time of 58.2 ms, 51.6% longer than the baseline’s 38.4 ms. The additional computation mainly comes from three sources: the self-attention calculations introduced by replacing ResNet-50 with the CLIP ViT-B/16 encoder, the similarity computations between regional features and 5000 attribute embeddings in the concept extender, and the Top-k retrieval with weighted aggregation. YOLO-World-L achieves the fastest speed with 168 GFLOPs and 19.2 ms inference time but attains only 27.6 APr, lower than our 28.5 on rare classes. Our approach surpasses YOLO-World-L by nearly 1 point in rare class performance, achieving a reasonable trade-off between accuracy and efficiency.

4.5. Robustness to Prompt Variations

To further evaluate the robustness of the proposed framework to variations in initial LLM instructions, we conducted experiments with three prompt variants. The Original prompts correspond to the four task-oriented templates used in our main experiments. We additionally designed a Concise prompts set, which provides shorter instructions, and a Detailed prompts set, which explicitly requests color, material, texture, shape, parts, and scale information from the LLMs.

Although the raw attributes generated by individual LLMs vary significantly across these prompt styles, the final attribute sets selected by our Game-Theoretic collaboration framework remain remarkably stable. As shown in

Table 10. Robustness Analysis under Different Prompt Configurations.

Prompt Setting	Concept Repository Size	Quality Score Q	Diversity Score D	RefCOCO	LVIS APr
Original prompts	5000	76.8	2.95	54.8	28.5
Concise prompts	5000	75.9	2.87	54.4	28.2
Detailed prompts	5000	77.1	2.91	54.9	28.4
Mean ± Std	—	76.6 ± 0.62	2.91 ± 0.04	54.7 ± 0.26	28.37 ± 0.15

, the downstream performance fluctuates only within a very narrow margin (LVIS APr variance is merely ±0.15). This robust stability arises from two core design choices: (1) the utility function evaluates attributes in the frozen CLIP semantic space considering quality, diversity, and redundancy, thereby filtering out prompt-induced linguistic noise; and (2) the cooperative Nash equilibrium ensures that no single prompt-sensitive LLM dominates the final selection. The results confirm that the framework is robust to moderate prompt variations, consistently maintaining a high-quality and effective Concept Repository.

5. Conclusions

This work addresses the core limitation of category-name-only semantic modeling in open-vocabulary object detection by introducing a cooperative multi-LLM game-theoretic Concept Expander. Treating attribute selection as a cooperative Nash equilibrium problem allows heterogeneous LLMs to jointly contribute complementary priors, systematically avoiding the coverage gaps and distribution biases of any single model. The resulting Concept Repository, injected via top-k retrieval and residual fusion into region-level features, consistently improves rare-category recognition, open-vocabulary generalization, and attribute-conditioned localization across multiple benchmarks. Gains are most pronounced for visually grounded attributes such as color, material, and texture, while spatially relational and negation-based reasoning remain open challenges. In future work, we plan to extend the current text-based concept repository toward a multimodal concept memory by incorporating visual exemplars, cross-modal relational cues, and online attribute updating. We expect this extension to improve attribute-aware grounding in broader multimodal scenarios, such as vision-language interaction, embodied perception, and interactive referring tasks.