Abstract
Visual Relationship Prediction (VRP) is crucial for advancing structured scene understanding, yet existing methods struggle with ineffective multimodal fusion, static relationship representations, and a lack of logical consistency. To address these limitations, this paper proposes a Hierarchical CLIP model (H-CLIP) for structured scene parsing. Our approach leverages a pre-trained CLIP backbone to extract aligned visual, textual, and spatial features for entities and their union regions. A multi-head self-attention mechanism then performs deep, dynamic multimodal fusion. The core innovation is a consistency and reversibility verification mechanism, which imposes algebraic constraints as a regularization loss to enforce logical coherence in the learned relation space. Extensive experiments on the Visual Genome dataset demonstrate the superiority of the proposed method. H-CLIP significantly outperforms state-of-the-art baselines on the predicate classification task, achieving a Recall@50 score of 64.31% and a Mean Recall@50 of 36.02%, thereby validating its effectiveness in generating accurate and logically consistent scene graphs even under long-tailed distributions.
1. Introduction
Visual Relationship Prediction (VRP) unlocks the true understanding in Structured Scene Parsing by modeling how objects interact, transforming simple detection into rich, relational scene comprehension. This capability is crucial for advancing applications that require deep scene understanding, content-aware image retrieval, and natural human-computer interaction [1]. However, conventional approaches are often hampered by three interrelated challenges: the ineffective fusion of multimodal information, the inflexible and static representation of relationships, and a pronounced lack of global logical consistency. These limitations constrain the generalization capability of models in complex, real-world scenarios and ultimately compromise predictive accuracy.
Existing VRP methods, particularly those built on deep learning frameworks, have made significant progress, like Faster R-CNN [2,3,4], and have been enhanced with relational reasoning modules, such as VTransE [5] and MOTIF-NET [6]. However, they encounter substantial limitations when parsing intricate scenes. Traditional models often rely on naive operations, like feature concatenation or average pooling, to fuse visual and textual data, failing to capture the complex, nonlinear interdependencies between modalities. Furthermore, by modeling relationships as static translations or treating each relational triplet as an independent classification instance, these methods lack the flexibility to adapt to the diverse visual manifestations of the same predicate and ignore the intrinsic logical connections among multiple relationships within a single scene [7,8]. Consequently, models may produce locally correct but globally inconsistent predictions, which undermine the structured scene understanding.
These shortcomings stem from an inability to effectively model relational dynamics and contextual dependencies, alongside underutilizing the complementary strengths of multimodal data. The bottlenecks manifest in three core aspects:
- (1)
- Ineffective Multimodal Fusion: Simple fusion strategies cannot dynamically integrate visual appearance, spatial layout, and semantic concepts, which is essential for disambiguating relationships across different contexts.
- (2)
- Inflexible Relational Representation: Static representation paradigms cannot adapt to the significant visual and spatial variations in the same relationship (e.g., “on” in “cup on table” vs. “cat on sofa”), harming generalization, especially for long-tail predicates.
- (3)
- Lack of Structural Reasoning Consistency: Most models lack explicit mechanisms to enforce logical consistency between predicted relationships in a scene, leading to contradictory outcomes.
In response to these challenges, this paper proposes a hierarchical framework of the CLIP model for visual relationship prediction (H-CLIP). Our framework is designed for Structured Scene Parsing, moving beyond independent prediction towards coherent graph construction. H-CLIP addresses the aforementioned bottlenecks through three key innovations:
- ▪
- A hierarchical feature embedding mechanism leverages a pre-trained CLIP model to extract and align deep visual, textual, and spatial features for entities and their union regions, which establishes a rich multimodal foundation.
- ▪
- An adaptive multimodal fusion strategy is implemented via a multi-head self-attention module, which enables deep, dynamic interaction between modalities rather than static concatenation.
- ▪
- A consistent and reversible relationship verification mechanism is given. This novel component introduces an auxiliary regularization loss based on an algebraic reversibility constraint, which guides the model to learn a relation feature space with an inherent logical structure, thereby enhancing global consistency and robustness.
Extensive experiments on the Visual Genome dataset demonstrate that our hierarchical framework significantly outperforms state-of-the-art benchmarks on the predicate classification (PredCls) task, achieving an R@50 score of 64.31%. Ablation studies confirm the individual contributions of our fusion strategy and consistency mechanism.
2. Related Works
Visual relationship prediction aims to identify the semantic and spatial relationships between pairs of objects in an image, which represents a crucial step from simple object recognition towards deep and structured scene understanding [9]. Our work builds upon and addresses limitations across several key research strands. This section reviews related work and categorizes it into three primary paradigms: (1) relational reasoning based on feature interaction and context aggregation, (2) relational modeling based on embedding space translation, and (3) the emerging paradigm leveraging visual-language models.
2.1. Relational Reasoning via Feature Interaction and Context Aggregation
This mainstream paradigm infers relationships by aggregating contextual features from subjects, objects, and the scene. Early models independently extracted visual features and used shallow networks or message passing for prediction [3,10,11,12]. To better utilize scene information, subsequent research adopted more sophisticated context aggregation mechanisms. MOTIF-NET [6] systematically introduced the scene graph concept, which used bidirectional LSTMs to propagate context and leveraged statistical co-occurrence priors (motifs). Following this, models based on Graph Neural Networks (GNNs) or tree structures, such as VCTree [7] and HGAT [13], further advanced the field by aggregating information over structured topologies.
More recently, Transformer-based architectures have attempted to solve scene graph generation as a set prediction problem, which utilized self-attention to directly output sparse triplets. However, these methods encounter substantial limitations when parsing intricate scenes. While RelTR [14] successfully captures global dependencies, its fixed number of queries restricts performance in extremely dense scenes where interactions exceed the query limit. Furthermore, the decoupled attention mechanisms may weaken fine-grained local interactions. Most critically, these data-driven approaches typically lack explicit mechanisms to enforce logical consistency.They often treat relationships as independent classification instances, ignoring intrinsic logical connections, which can lead to locally correct but globally inconsistent predictions.
2.2. Relational Modeling Based on Embedding Space Translation
Inspired by knowledge graph embedding techniques, like TransE [15], this paradigm maps entities and relationships into a unified vector space. VTransE [5] pioneered this direction in VRP, modeling a predicate as a translation vector such that . Its successor, UVTransE [16], enhanced representation by incorporating visual features from the union box.
The core strength of translation-based models is their geometric interpretability. However, their primary bottleneck lies in the static nature of the representation. A fixed translation vector is learned for each predicate, which fails to adapt to the vast visual and spatial diversity of the same relationship across different instances (e.g., the visual appearance of “riding” varies significantly between “person riding bike” and “child riding horse”). Additionally, simple vector translation struggles to model complex relationship patterns, such as one-to-many interactions. Our approach addresses this by introducing dynamically predicted weights, injecting the translation model with the flexibility to adapt to diverse visual manifestations.
2.3. The Emerging Visual-Language Model Paradigm
The rise of large-scale pre-trained models has introduced a powerful new paradigm. The strong cross-modal semantic alignment of CLIP [17], learned from vast image-text pairs, provides a shared space for processing visual and textual data, offering new opportunities for VRP, particularly in open-vocabulary settings [18,19,20].
A central challenge is how to effectively fuse aligned features from different modalities to unlock the full potential of visual-language model [21]. Common strategies of simple concatenation often fail to capture complex cross-modal interactions. More importantly, recent analyses suggest that while visual-language models excel at coarse-grained matching, they struggle with structured logic, such as fine-grained spatial relations or word ordering [22,23]. Merely extracting features from the AV frozen model without enforcing structural constraints fails to filter out implausible predictions. Our H-CLIP framework bridges this gap by integrating the semantic power of CLIP model with a rigorous algebraic consistency verification mechanism, which can ensure that the learned relation space adheres to geometric logic.
Therefore, our H-CLIP framework synthesizes and significantly advances insights from these paradigms. It leverages the contextual awareness of graph-based models, which dynamically extends the translational embedding concept and deeply integrates the cross-modal power of visual-language models. The key novelty lies in the three-fold integration of: (1) hierarchical CLIP-based feature extraction for entities and union regions, (2) dynamic multimodal fusion via attention, and (3) a novel algebraic consistency and reversibility regularizer. This integrated approach addresses the core limitations of prior work, which typically incorporates one or two of these components.
3. Problem Definition and Framework
This section first addresses the problem of structured scene parsing, and an advanced task aims to generate a graph-based representation of an image, which captures not only objects but also their semantic and spatial relationships (e.g., riding, on top of). Then, a hierarchical CLIP Model is given.
3.1. Problem Definition
Structured Scene Parsing is an advanced computer vision task, that builds upon semantic/instance segmentation. It aims to not only identify and delineate individual objects in an image but also to explicitly model the relationships (e.g., spatial, semantic, functional, interactive) between these objects. The goal is to generate a structured representation (often as a graph) of the scene that captures its deeper semantics, such as object interactions, scene functionality, and holistic context.
In the problem of structured scene parsing, the input is a digital image, denoted as I ∈ , where H, W and C represent the image height, width, and number of channels, respectively. The output is a structured scene graph G = , which is a labeled, directed graph representing the deep semantic structure of the image. It explicitly frames the generation of a structured knowledge graph as the ultimate objective of the Structured Scene Parsing task. This integration successfully establishes the overarching problem context and desired output before introducing the solution.
The node set V of G = represents the semantic entities (objects or regions) in the image. Each node ∈ V corresponds to an instance in the image and each node is associated with an attribute tuple = . Here, denotes the semantic category of the entity (e.g., “person”, “car”, “table”) from a predefined category set , satisfying ∈, represents the spatial bounding box of the entity, typically represented as , satisfying ∈ and ∈ refers to the binary mask of the entity, precisely delineating its shape at the pixel level. This is the core feature that connects this task to semantic/instance segmentation, going beyond traditional object detection.
The edge set E of G = represents the meaningful semantic or spatial relationships between pairs of entities. Each directed edge = ∈ E goes from node (subject) to node (object) and each edge is associated with a relationship label = ∈, where belongs to a predefined set of relationship predicates . This set typically includes: Spatial relations:“on top of”, “to the left of”, “inside”, etc. Action/Interaction relations: “riding”, “holding”, “looking at”, “chasing”, etc. Comparative relations: “larger than”, “darker than”, etc. Semantic relations: “part of”, “belonging to”, etc.
The objective of the task is to find the optimal scene graph that maximizes the following joint probability:
In practice, this complex joint probability is typically factorized and modeled as a combination of three subtasks:
- (1)
- Entity Perception: is used to identify what entities are present in the image and determining their category, location, and mask.
- (2)
- Relationship Existence Judgment: is used to determine between which entity pairs meaningful relationships exist (i.e., graph construction).
- (3)
- Relationship Classification: is used to assign a specific predicate label to each existing edge (relationship).
For the above subtasks, the hierarchical model based on CLIP (H-CLIP model for short) is proposed to capture the deeper semantics of visualized images. For the subtask (1) of Entity Perception, the Faster CNN is used to identify the bounding boxes of entities in the original image, and the entities are classified and represented by different colored rectangular boxes. For the subtask (2) of Relationship Existence Judgment, the H-CLIP model calculates the inclusion relationship between entity bounding boxes in a top-down manner, where high-level entity bounding boxes contain low-level entity bounding boxes. For the subtask (3) of Relationship Classification, the H-CLIP model constructs the relational semantics between entity bounding boxes in a bottom-up manner. Finally, the deeper semantics are captured, and a knowledge graph is constructed based on the entity and relational semantics of the entity bounding box.
A Framework for Constructing Image Generative Knowledge Graph Based on Hierarchical Model is shown in Figure 1. The pipeline begins with Entity Perception using Faster R-CNN to detect and classify objects, which are represented by colored bounding boxes. Relationship Existence Judgment identifies potential connections between entity pairs. Finally, Relationship Classification uses the H-CLIP model to predict specific predicates, forming as knowledge fragments, where , denote entity masks, and , denote predicate labels. These fragments are integrated to construct the final structured Knowledge Graph. The figure has been designed for clarity to illustrate the hierarchical flow from visual input to structured output.
Figure 1.
A Framework for Constructing Image Generative Knowledge Graph Based on Hierarchical Model.
3.2. Overall Framework of H-CLIP
This section presents the overall framework of the proposed Hierarchical CLIP Model (H-CLIP), a comprehensive pipeline designed for structured scene parsing. The primary objective is to transform an input image into a logically consistent set of visual relationships, forming the foundation for structured knowledge graph construction. The framework operates through five sequential stages, as illustrated in Figure 2.
Figure 2.
Overall Framework of H-CLIP.
Stage 1: Entity Detection. The process begins with entity detection and classification using a pre-trained Faster R-CNN. This model identifies and localizes all object instances within the input image, providing for each entity its bounding box coordinates and a preliminary class label (e.g., “person”, “bicycle”). These detected entities form the fundamental subject and object candidates for relationship prediction.
Stage 2: Hierarchical CLIP Extraction. For each candidate subject-object pair, the framework performs hierarchical feature extraction using a frozen CLIP model. The union bounding box is defined that minimally encloses the subject and object as a higher-level contextual region. The text encoders of CLIP are then used to extract aligned, multimodal feature embeddings for three key regions: the subject (s), the object (o), and their union (u). This yields visual features (, , ) and corresponds to the textual semantic features, establishing a rich, semantically grounded representation at both the entity and contextual levels.
Stage 3: Dynamic Fusion. The extracted visual, textual, and separately encoded spatial features for the subject, object, and union regions are fed into a dynamic multimodal fusion module. Instead of simple concatenation, this module employs a Multi-Head Self-Attention (MHSA) mechanism. This allows the model to dynamically learn and weight the complex interactions and dependencies between different modalities (appearance, language, space), generating a robust, context-aware fused feature representation for each of the three regions (, , ).
Stage 4: Relation Modeling. The fused features are used for relation prediction via a dynamically weighted translational model. Inspired by knowledge graph embeddings but significantly extended, a discriminative relation feature is computed as = − − . The weights and are predicted dynamically based on the visual context, allowing the model to adaptively filter out entity-specific information from the union context to isolate the pure relational semantics. This feature is then classified into a specific predicate label (e.g., “riding”).
Stage 5: Output & Consistency Verification. To enforce logical coherence among predictions, a consistency verification mechanism is applied as an auxiliary training objective. It introduces a novel reversibility constraint loss function (), which regularizes the learned feature space by requiring that the combined representation of multiple relationships in a scene be decomposable into the sum of their constituent entity contributions. This algebraic regularization guides the model to learn structurally consistent and reversible relation representations, significantly improving generalization.
The final output is a set of verified visual relationship triplets (subject, predicate, object), which can be directly assembled into a structured scene graph. This modular and hierarchical pipeline ensures both accurate and logically coherent structured scene parsing.
4. Our Framework of Hierarchical CLIP Model
This section introduces the Hierarchical CLIP (H-CLIP) framework for constructing a structured knowledge graph from images. The framework integrates vision, language, and spatial cues in a hierarchical manner to enable explainable and structured visual relationship understanding.
4.1. Entity Detection and Classification
The framework begins with precise localization and recognition of visual entities in the input image. A pre-trained Faster R-CNN model with frozen parameters is employed to detect all object instances. For each detected entity (subject s or object o), the model provides its bounding box, defined by its top-left coordinates, width, and height ( = ), and its class label (). These bounding boxes, often visualized with distinct colors, form the fundamental visual units and the initial nodes for the subsequent knowledge graph construction. The process transforms raw pixel data into structured entity proposals, establishing the “what” and “where” of the visual scene.
4.2. Top-Down Inclusion Relationship Calculation
The hierarchy in H-CLIP is established through the computation of a union region for each subject-object pair. This union box serves as the higher-level contextual region that spatially contains the individual subject () and object () boxes. The model evaluates this spatial containment to establish a contextual hierarchy for each relational triplet, rather than building a global, multi-layer tree of all entities. In this framework, top-down refers to the process of forming this joint contextual representation () from the individual entities, while bottom-up describes the subsequent flow of refined, context-aware information from back to inform the relation prediction for the (, ) pair.
Building upon the detected entities, the framework establishes a spatial-semantic hierarchy through a top-down analysis of containment relationships. For any pair of subject s and object o, their union box = ∪ is computed, representing the minimal enclosing rectangle containing both entities. This union box acts as a higher-level contextual region. The model systematically evaluates bounding boxes to determine if a higher-level entity’s region (like ) spatially contains a lower-level entity’s region (like or ). This step formalizes part-whole hierarchies (e.g., a “car” box contains “wheel” and “door” boxes) and is fundamental for structuring entities from coarse to fine granularity. The spatial coordinates of , and are projected into a learned embedding space via linear functions , and : → to obtain spatial features , and .
4.3. Entity Semantic Identification at Each Level
At each level of the established hierarchy (individual entities and their union regions), the model performs rich multimodal semantic grounding. This involves extracting both visual and textual feature embeddings aligned in a shared semantic space.
Visual Feature Encoding: The image patches corresponding to the subject region , object region and union region are fed into a frozen CLIP Image Encoder. This yields dense visual embeddings:
Textual Feature Encoding: Descriptive text prompts are constructed for each region. For subject s and object o, prompts are based on their class labels and . For the union region u, a contextual prompt is formed by combining the subject description , object description , and a potential relational predicate . These textual prompts are encoded by the CLIP Text Encoder:
These steps translate visual entities into semantically meaningful vector representations (, , etc.) at different hierarchical levels, which is conducted to capture both their appearance and conceptual meaning.
4.4. Bottom-Up Relational Semantic Construction
The final phase synthesizes the individual entity semantics into explicit relational predicates through a bottom-up feature fusion and reasoning process.
4.4.1. Multimodal Feature Fusion
For each entity/region (j ∈ ), its visual (), textual (), and spatial () features are first projected into a unified hidden space via modality-specific linear layers with LayerNorm and ReLU activation (Formula (4)).
Here, presents the modal m (such as visual, text or coordinate) feature of entity j (such as subject s, object o or joint region u), and is the projection weight corresponding to the corresponding mode. The projected features are then processed by a Multi-Head Self-Attention (MHSA) mechanism (Formulas (5) and (6)).
where , , and are the first i learnable weight matrices, and is the dimension of the key vector. Then, all attention head outputs are concatenated and passed through a final linear projection to the output of the multi-head attention mechanism, shown in Formula (6).
This allows the model to dynamically learn interactions and dependencies across different modalities. The MHSA output is averaged across the modality dimension and passed through a Deep MLP with residual connections to produce a robust, fused entity feature . The multi-head attention mechanism dynamically captures complex interactions and dependencies between multimodal features, thereby generating more comprehensive and expressive feature representations. To aggregate the multimodal attention outputs (dimensional ) into a single fused feature, an average across the modalities is shown in Formula (7).
To further enhance the model’s expressive power and feature robustness, a DeepMLP (Multi-Feature Learning Path) is introduced, which consists of multi-layer linear transformations, ReLU activation, LayerNorm, and Dropout. The fused features undergo these deep layers, incorporating residual connections, shown in Formulas (8) and (9).
Finally, by aligning the feature dimensions through linear projection, the final fused entity features is obtained. After completing the multimodal feature fusion across key regions (subject, object, and joint regions) using a multi-head attention mechanism, this results in fused features for the subject (), object () and union region ().
4.4.2. Dual-Stream Relation Feature Formulation and Prediction
To address the contradiction between geometric interpretability and semantic generalization, the H-CLIP framework employs a dual-stream architecture comprising an “Explicit Geometric Decoupling Stream” and an “Implicit Semantic Perception Stream”.
First, the Explicit Stream aims to isolate “pure” relational semantics by performing a geometric “reverse decoupling” in the feature space. It subtracts the contributions of the subject and object entities from the union context using dynamically learned weights and , as shown in Formula (10):
where and are dynamic weights generated by a lightweight MLP based on the context, satisfying .
Simultaneously, the Implicit Stream utilizes a non-linear Multi-Layer Perceptron (MLP) to directly extract high-level semantic context from the union features, capturing complex non-linear relationships that the explicit stream might miss:
To incorporate scene-level priors and eliminate ambiguity, a Global Context Gating mechanism is applied. The final relation feature is obtained by gating the explicit feature with a channel-wise coefficient derived from the global image feature, as described in Formula (12):
Finally, a classifier head uses to predict the probability distribution over a predefined set of predicates , optimized with a standard cross-entropy loss :
where C is the number of categories, y is the one-hot ground truth, and is the predicted probability.
4.4.3. Dual-Stream Consistency Validation Mechanism
To bridge the gap between geometric logic and visual perception, an auxiliary consistency verification mechanism is introduced. As detailed in the “consistency of semantic mapping” principle, the explicit stream provides structured geometric priors, while the implicit stream provides rich contextual semantics. In an ideal feature space, both streams should point to the same consistent relational semantics.
Therefore, we introduce a Dual-Stream Consistency Loss to enforce the alignment between the explicit feature and the implicit feature . This is formulated as the Mean Squared Error (MSE) between the two representations:
Here, N represents the batch size, and denotes the norm. Minimizing this loss serves a dual purpose: it forces the dynamic weights in the explicit stream to reflect true semantic contributions, and explicitly constrains the implicit stream with geometric logic to prevent overfitting to background noise.
The final objective function is a weighted sum of the classification loss and the consistency loss, controlled by the hyperparameter :
To further elucidate the rationale behind the dual-stream design, we consider a typical interaction scenario, such as “Person riding Horse”. In the Explicit Stream, the model performs a “geometric decoupling” operation in the feature space: it mathematically subtracts the feature embeddings of the subject (“Person”) and the object (“Horse”) from the holistic union region (). Theoretically, the remaining feature residual should strictly isolate the visual cues specific to the interaction itself (e.g., the straddling posture or the contact point on the saddle), representing the “pure” relational logic. Conversely, the Implicit Stream utilizes the non-linear inference capability of neural networks to directly extract the high-level semantic context of “riding” from the union appearance without explicit separation. The proposed consistency verification () enforces that the relation feature derived from “logical subtraction” aligns with the one derived from “contextual perception”. This constraint prevents the model from relying solely on statistical biases (e.g., co-occurrence of person and horse) and ensures that the predicted relationship is grounded in both geometric structural logic and semantic consistency.
Experimental results demonstrate that this internal logic constraint significantly improves the model’s robustness and generalization ability, particularly for long-tailed distributions and complex scenarios.
In summary, our Hierarchical CLIP Model (H-CLIP) systematically constructs a robust visual relation detector. By integrating the “Explicit Geometric Decoupling” and “Implicit Semantic Perception” via the consistency regularization, H-CLIP achieves a unified representation of geometric logic and neural perception.
5. Experimental Results
This section evaluates our model on the Predicate Classification (PredCls) task. Under this setting, the model takes as input an image together with the ground-truth bounding boxes and object category labels. The objective is to predict the semantic relationship (i.e., predicate) for each pair of objects in the scene.
5.1. Experimental Setup
Dataset and Split Protocol. The proposed H-CLIP framework is evaluated on the Visual Genome (VG) dataset, which is the standard benchmark for visual relationship prediction. Following the widely adopted VG150 split protocol [10], the dataset is filtered into the most frequent 150 object categories and 50 predicate categories. This filtering strategy ensures a balance between sample sufficiency and category diversity, covering over 75% of the relationship triplets in the original data. The dataset comprises 108,077 images. In accordance with the standard protocol, the dataset is divided into a training set (approximately 70% of images) and a testing set (approximately 30% of images). A validation set of 5000 images is sampled from the training set for hyperparameter tuning. On average, the dataset provides a dense scene representation with approximately 21 objects and 18 relationships per image.
Task Definition. this paper focuses on the Predicate Classification (PredCls) task, which is conducted to isolate the performance of the relationship prediction module from object detection errors. In the PredCls setting, the model is provided with the ground-truth bounding boxes and object category labels for all entities in the image. The goal is to predict the correct predicate label describing the interaction between each pair of objects.
Implementation Details. For the H-CLIP components, the pre-trained CLIP (ViT-B/32) model is employed. The visual and textual encoders of CLIP are used to extract aligned embeddings for entity regions, union regions, and label descriptions. The model is implemented using PyTorch 1.12.1. All multimodal features are mapped to a shared hidden dimension of before the multi-head attention fusion. The consistency loss weight is set to based on ablation studies. The model is trained using the AdamW optimizer with an initial learning rate of and a weight decay of . The batch size is set to 32, and the model is trained for 50 epochs with a linear learning rate warmup for the first epoch.
5.2. Visual Relationship Prediction
For performance evaluation, the widely used Recall@K (R@K) metric is adopted. It calculates the fraction of ground-truth relationship triplets that appear among the top-K most confident predictions per object pair. To provide a thorough assessment, the results are reported across several thresholds: R@5, R@10, R@20, and R@50.
To comprehensively benchmark our H-CLIP model, it is compared through a range of representative and state-of-the-art visual relationship prediction methods.
Table 1 presents the overall recall performance on the Visual Genome dataset, where the bold number is used to highlight the results of our method. The proposed H-CLIP framework, particularly when equipped with the consistency loss (H-CLIP + inv Loss), achieves competitive results against state-of-the-art methods. Specifically, H-CLIP + inv Loss attains an R@50 of 64.31%, outperforming the strong Transformer-based baseline RelTR (63.10%) and the graph-based HGAT (59.40%).
Table 1.
Comparison of results of VG dataset. The bold number is used to highlight the results of our method.
A critical observation regarding prediction quality can be made by comparing our model with VCTree + TDE + HIERCOM. While the baseline achieves a high R@100 (67.50%), its performance drops significantly at R@50 (58.30%). This large gap indicates that the baseline relies on generating a large number of candidates to cover the ground truth, struggling to rank correct predictions at the top. In contrast, H-CLIP maintains a consistent performance (64.31% for both R@50 and R@100), demonstrating its superior ability to prioritize correct relationships with high confidence and logical stability.
To further validate the effectiveness of multimodal fusion strategies in the H-CLIP model, various common feature fusion methods are compared. Specifically, in the model of the feature fusion module, the methods are selected, such as splicing, gating mechanisms, and attentional mechanisms, for substitution in comparative analysis experiments, which are conducted to observe changes in model performance. The experimental results are shown in Table 2.
Table 2.
Comparison of results of VG dataset on meanRecall. The bold number is used to highlight the results of our method.
To evaluate performance on the long-tailed distribution of relationships, the Mean Recall (mR@K) is analyzed in Table 2. This metric treats all relationship categories equally and prevents dominance by frequent classes. The results show that H-CLIP + inv Loss achieves the best performance across all metrics, with an mR@50 of 36.02% and mR@100 of 36.67%. Two key conclusions can be drawn from these results:
Superiority over Statistical De-biasing: The VCTree + TDE method incorporates the Total Direct Effect (TDE) specifically for de-biasing tail classes. However, our H-CLIP + inv Loss surpasses it by approximately 4% on mR@50 (36.02% vs. 32.09%). This suggests that enforcing structural logical consistency in the feature space is more effective for transfer learning to rare classes than post-hoc statistical calibration.
Effectiveness of Consistency Constraint: Introducing the dual-stream consistency verification mechanism yields a substantial performance leap. Compared to the base H-CLIP model, adding the inv Loss boosts mR@50 from 32.14% to 36.02%. This validates that the differentiable logic constraint forces the model to learn the intrinsic “geometric logic” of relationships rather than rote memorization, thereby significantly enhancing robustness in complex or data-scarce scenarios.
5.3. Ablation Experiment
To determine the optimal contribution of the consistency constraint, we conducted a sensitivity analysis on the hyperparameter (denoted as in Formula (14)). This parameter balances the trade-off between the classification loss and the consistency loss .
As shown in Table 3, the performance exhibits a trend of increasing and then slightly decreasing:
Table 3.
Ablation study on hyperparameter .
Baseline (): With , the model operates in a pure explicit stream mode without logical verification. The R@50 of 63.52% reflects the lower bound performance relying solely on geometric structural features.
Optimal Balance (): Introducing the implicit stream as a regularizer significantly improves performance. The accuracy peaks at (R@50: 64.31%), indicating that the model achieves an optimal equilibrium between geometric logical decoupling and semantic consistency constraints. The consistency loss effectively filters out noisy predictions that contradict the contextual logic.
Over-Regularization (): When increases to 0.6 or 0.8, a slight performance drop is observed. This suggests that excessive regularization may restrict the explicit stream’s ability to fit complex, non-linear scenarios, sacrificing some feature flexibility for strict algebraic consistency. Consequently, is adopted for all subsequent experiments.
To validate the effectiveness of the proposed deep multimodal fusion, our multi-head self-Attention mechanism is compared against other fusion strategies, which represents an evolution from simple interaction to deep interaction. The results in Table 4 reveal the following:
Table 4.
Ablation study on feature fusion strategies. The bold number is used to highlight the results of our method.
Concatenation (Baseline): Simple feature concatenation yields the lowest performance (R@50: 62.29%). This physical stacking fails to bridge the heterogeneity gap between visual, textual, and spatial modalities, limiting the capture of non-linear associations.
Gating & Single-Head Attention: Both mechanisms improve over concatenation by introducing adaptivity. The Gating Mechanism (63.72%) highlights the importance of “adaptive weighting” to suppress noise. However, Single-Head Attention (63.60%) performs comparably to gating, as a single attention head struggles to simultaneously attend to multiple distinct aspects of the feature space (e.g., focusing on both spatial geometry and semantic co-occurrence).
Multi-Head Attention (Ours): The proposed approach achieves the best performance (R@50: 64.31%), which surpasses the concatenation baseline by over 2%. This validates the design philosophy that multi-head attention enables a division of labor. It allows the model to project features into parallel subspaces, where different heads can independently model geometric relations and semantic associations, which results in a richer and more robust representation.
6. Conclusions
This paper addresses core challenges in Visual Relationship Prediction (VRP), including effective multimodal fusion, flexible relational representation, and structurally consistent reasoning. To tackle these, we propose a novel Hierarchical CLIP framework for Prediction (H-CLIP).
Our framework first leverages CLIP to extract aligned visual, textual, and spatial features. A multi-head self-attention module then dynamically fuses these modalities. The core innovation is a dynamic-weighted relational predictor coupled with a consistency and reversibility verification mechanism, which applies algebraic constraints via a regularization loss to enhance logic, robustness, and generalization.
Extensive experiments on the Visual Genome dataset show that H-CLIP outperforms state-of-the-art methods on key metrics. Ablation studies confirm the contribution of each module, especially the attention fusion and consistency loss.
Limitations include the model’s relatively high computational cost, which may affect real-time use, and room for improvement in handling complex one-to-many and many-to-many relationships. Furthermore, the current experimental evaluation is conducted primarily on the PredCls task; future work may apply and evaluate the framework on the full suite of scene graph generation tasks (SGCls and SGDet). The reliance on standard Recall@K also calls for complementary analysis using metrics like mean Recall@K to fully assess performance on rare predicates. Future work will focus on improving efficiency and exploring more effective fusion strategies for complex relational reasoning.
Author Contributions
Conceptualization: Y.S.; Methodology: Y.L.; Validation: R.Q.; Software: X.C.; Writing—Original Draft Preparation: Y.S.; Writing—Review & Editing: R.Q. and Y.S.; Supervision: R.Q. and H.C.; Funding Acquisition: R.Q., H.C. and Y.S. Authorship is limited to those who have made substantial contributions to the reported work. All authors have read and agreed to the published version of the manuscript.
Funding
This work was supported by the Basic Research Foundation of Liaoning Province (No. LJK QZ20222436, No. 2022J H2/101300270), the Liaoning Provincial Natural Science Foundation (No. 2024-MS-174), the General Project of the Humanities and Social Sciences Planning Fund of the Ministry of Education (No. 23YJAZH109) and the Liaoning Provincial Department of Education’s 2024 Basic Research Program for Higher Education Institutions under the project “Conditional Generation-Based Fusion of Imbalanced Multimodal Comment Data” (No. LJ212410172022).
Data Availability Statement
Data is contained within the article.
Conflicts of Interest
The authors declare no conflicts of interest. The code and detailed documentation are available at: https://github.com/shawnBlue957/Repository-for-HMEP (accessed on 6 January 2026). All datasets used are publicly accessible via the links and DOI provided.
References
- Neau, M.; Santos, P.; Bosser, A.G.; Buche, C. In Defense of Scene Graph Generation for Human-Robot Open-Ended Interaction in Service Robotics. In Robot World Cup; Springer: Cham, Switzerland, 2023; pp. 299–310. [Google Scholar]
- Girshick, R. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1440–1448. [Google Scholar]
- Dai, J.; He, K.; Sun, J. Instance-Aware Semantic Segmentation via Multi-Task Network Cascades. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 3150–3158. [Google Scholar]
- Zareian, A.; Karaman, S.; Chang, S.-F. Weakly Supervised Visual Semantic Parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 3736–3745. [Google Scholar]
- Zhang, H.; Kyaw, Z.; Chang, S.-F.; Chua, T.-S. Visual Translation Embedding Network for Visual Relation Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 5532–5540. [Google Scholar]
- Zellers, R.; Yatskar, M.; Thomson, S.; Choi, Y. Neural Motifs: Scene Graph Parsing with Global Context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 5831–5840. [Google Scholar]
- Tang, K.; Zhang, H.; Wu, B.; Luo, W.; Liu, W. Learning to Compose Dynamic Tree Structures for Visual Contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 6619–6628. [Google Scholar]
- Lu, P.; Peng, B.; Cheng, H.; Galley, M.; Chang, K.-W.; Wu, Y.N.; Zhu, S.-C.; Gao, J. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. Adv. Neural Inf. Process. Syst. 2023, 36, 43447–43478. [Google Scholar]
- Johnson, J.; Krishna, R.; Stark, M.; Li, L.-J.; Shamma, D.; Bernstein, M.; Li, F.-F. Image Retrieval Using Scene Graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3668–3678. [Google Scholar]
- Xu, D.; Zhu, Y.; Choy, C.B.; Li, F.-F. Scene Graph Generation by Iterative Message Passing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 5410–5419. [Google Scholar]
- Li, Y.; Ouyang, W.; Zhou, B.; Wang, K.; Wang, X. VIP-CNN: Visual Phrase Guided Convolutional Neural Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 1347–1356. [Google Scholar]
- Lu, C.; Krishna, R.; Bernstein, M.; Li, F.-F. Visual Relationship Detection with Language Priors. In Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, 11–14 October 2016; pp. 852–869. [Google Scholar]
- Mi, L.; Chen, Z. Hierarchical Graph Attention Network for Visual Relationship Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 11187–11196. [Google Scholar]
- Cong, Y.; Yang, M.Y.; Rosenhahn, B. Reltr: Relation Transformer for Scene Graph Generation. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 11169–11183. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; Yakhnenko, O. Translating Embeddings for Modeling Multi-Relational Data. Adv. Neural Inf. Process. Syst. 2013, 26, 2787–2795. [Google Scholar]
- Hung, Z.-S.; Mallya, A.; Lazebnik, S. Contextual Translation Embedding for Visual Relationship Detection and Scene Graph Generation. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 43, 3820–3832. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Radford, A.; Kim, J.W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the International Conference on Machine Learning (ICML), Virtual, 18–24 July 2021; Proceedings of Machine Learning Research. pp. 8748–8763. [Google Scholar]
- Subramanyam, R.; Jayram, T.S.; Anirudh, R.; Thiagarajan, J.J. Exploring the Utility of Clip Priors for Visual Relationship Prediction. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea, 14–19 April 2024; pp. 6825–6829. [Google Scholar]
- Mitra, C.; Huang, B.; Darrell, T.; Rohrbach, A. Compositional Chain-of-Thought Prompting for Large Multimodal Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 16–22 June 2024; pp. 14420–14431. [Google Scholar]
- Yu, Q.; Li, J.; Wu, Y.; Zhang, Y.; Wang, Y.; Wang, Y.; Yang, M. Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, 2–6 October 2023; pp. 21560–21571. [Google Scholar]
- Jiang, B.; Liu, X.; Wang, Y.; Taylor, C.J. Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 3–8 January 2025; pp. 1234–1243. [Google Scholar]
- Yuksekgonul, M.; Bianchi, F.; Kalluri, P.; Jurafsky, D.; Zou, J. When and Why Vision-Language Models Behave Like Bags-of-Words, and What to Do About It? arXiv 2022, arXiv:2210.01936. [Google Scholar] [CrossRef] [Scilit]
- Zhao, Y.; Shu, T.; Katz, B.; Barbu, A.; Mao, J. VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations. arXiv 2022, arXiv:2207.00221. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.

