ProM-Pose: Language-Guided Zero-Shot 9-DoF Object Pose Estimation from RGB-D with Generative 3D Priors

Abstract

Object pose estimation is fundamental for robotic manipulation, autonomous driving, and augmented reality, yet recovering the full 9-DoF state (rotation, translation, and anisotropic 3D scale) from RGB-D observations remains challenging for previously unseen objects. Existing methods either rely on instance-specific CAD models, predefined category boundaries, or suffer from scale ambiguity under sparse observations. We propose ProM-Pose, a unified cross-modal temporal perception framework for zero-shot 9-DoF object pose estimation. By integrating language-conditioned generative 3D shape priors as canonical geometric references, an asymmetric cross-modal attention mechanism for spatially aware fusion, and a decoupled pose decoding strategy with temporal refinement, ProM-Pose constructs metrically consistent and semantically grounded representations without relying on category-specific pose priors or instance-level CAD supervision. Extensive experiments on CAMERA25 and REAL275 benchmarks demonstrate that ProM-Pose achieves competitive or superior performance compared to category-level methods, with mAP of $75.0\%$ at $5^{°},2\mathrm{cm}$ and $90.5\%$ at ${10}^{°},5\mathrm{cm}$ on CAMERA25, and $42.2\%$ at $5^{°},2\mathrm{cm}$ and $76.0\%$ at ${10}^{°},5\mathrm{cm}$ on REAL275 under zero-shot cross-domain evaluation. Qualitative results on real-world logistics scenarios further validate temporal stability and robustness under occlusion and lighting variations. ProM-Pose effectively bridges semantic grounding and metric geometric reasoning within a unified formulation, enabling stable and scale-aware 9-DoF pose estimation for previously unseen objects under open-vocabulary conditions.

1. Introduction

With the rapid development of Embodied AI and intelligent robotic systems, enabling autonomous platforms to perceive and interact with previously unseen environments has become a key technological requirement. Object pose estimation provides the fundamental spatial understanding needed for robotic grasping [1,2,3,4], autonomous driving [5], and augmented reality [6]. In practical deployments, recovering not only object pose but also physically consistent object scale is critical for safe manipulation, collision avoidance, and reliable metric reasoning in both digital and physical interaction tasks. However, reliably estimating the full 9-DoF state (rotation, translation, and 3D scale) from real-world RGB-D observations remains challenging due to illumination variation, occlusion, and the requirement to generalize across previously unseen objects.

Existing methods predominantly operate at the instance level [7,8,9,10,11,12,13], relying on precise CAD models or instance-specific supervision to recover object pose. While these approaches achieve high accuracy under controlled settings, their deployment is inherently restricted to pre-scanned objects and fixed inventories. To overcome this limitation, category-level methods [14,15,16,17,18,19] aim to learn shared geometric representations across object classes, enabling intra-category generalization. However, such methods remain bounded by predefined category taxonomies and often suffer from performance degradation when faced with substantial intra-class scale variation or pronounced geometric diversity. As illustrated in Figure 1, neither paradigm adequately addresses scenarios in which object identities are unknown a priori or extend beyond predefined semantic boundaries.

To tackle these data scarcity and cross-domain generalization bottlenecks, intelligent methods based on deep learning have been widely adopted and achieved good results across various engineering domains. For instance, advanced transfer learning and domain adaptation techniques, such as conditional distribution-guided adversarial transfer learning networks [20], adaptive fused domain-cycling variational generative adversarial networks [21], and dynamic collaborative adversarial domain adaptation networks [22], have demonstrated remarkable success in effectively bridging domain gaps and synthesizing robust features under unlabeled or scarce-data conditions. Drawing inspiration from such cross-domain success, vision-based pose estimation has similarly embraced domain adaptation and generalization strategies to handle unseen environments and bridge the synthetic-to-real domain gap. Numerous cross-domain and domain-translation pipelines [23,24,25,26,27] have been proposed to align feature distributions or translate image styles between synthetic renderings and real-world scenes, attempting to generalize pose estimators to unannotated target domains.

Beyond these adaptation-based strategies, recent zero-shot paradigms have emerged to eliminate the reliance on instance-specific supervision altogether. Model-based foundation approaches [28,29] utilize generic geometric priors to alleviate dependency on category-specific training, whereas model-free registration pipelines [30,31,32,33,34] estimate pose from reference views without requiring explicit CAD models. Although these strategies relax supervision constraints, most existing solutions remain limited to 6-DoF pose estimation, assume isotropic or known object scale, or rely on sparse-view correspondences. Their formulations rarely incorporate explicit mechanisms for metric-scale observability or anisotropic scale recovery. For example, FoundationPose [35] demonstrates strong tracking robustness but assumes known object scale. Generative 9-DoF approaches such as the Diff9D [36] model pose distributions at the category level and depend on predefined class-specific training. Consequently, these methods still operate within fixed semantic boundaries, creating structural limitations when encountering previously unseen objects under open-vocabulary conditions.

Figure 1. A comparison of instance-level, category-level, and unseen methods is shown. Instance-level methods can only estimate the pose of specific object instances on which they are trained. Category-level methods can infer intra-class unseen instances rather than being limited to specific instances in the training data. In contrast, unseen object pose-estimation methods have stronger generalization ability and can handle object categories not encountered during training [37].

Figure 1. A comparison of instance-level, category-level, and unseen methods is shown. Instance-level methods can only estimate the pose of specific object instances on which they are trained. Category-level methods can infer intra-class unseen instances rather than being limited to specific instances in the training data. In contrast, unseen object pose-estimation methods have stronger generalization ability and can handle object categories not encountered during training [37].

Recent advances in Vision–Language Models (VLMs) [38,39] provide powerful open-vocabulary semantic priors that enable text-conditioned spatial reasoning, including language-guided localization [40,41] and instruction-driven manipulation [42,43,44]. Nevertheless, most VLM-driven approaches adopt a semantic matching paradigm that prioritizes object identification over metrically consistent geometric recovery. As a result, they often operate in normalized or view-dependent coordinate spaces, making physically meaningful scale estimation and temporally stable 9-DoF reasoning difficult without explicit geometric constraints. We argue that the fundamental bottleneck of zero-shot 9-DoF pose estimation lies in the structural disconnect between semantic selection, geometric consistency, and scale observability. Existing methods partially address one or two of these aspects, but they rarely unify semantic openness, geometric consistency, and metric-scale observability within a single formulation. Vision–language representations provide strong semantic grounding but limited depth-aware geometric reasoning, while geometry-driven methods offer physical interpretability yet remain unstable without robust semantic anchors in cluttered scenes. Under sparse-view conditions, the intrinsic coupling between rotation and scale further amplifies ambiguity in recovering real-world dimensions, undermining reliable spatial perception for downstream tasks.

To address these limitations, we propose ProM-Pose, a unified cross-modal temporal perception framework for zero-shot 9-DoF object pose estimation that explicitly bridges semantic grounding and metric geometric reasoning under open-world conditions. Unlike conventional pipelines, ProM-Pose does not learn category-dependent pose distributions. Instead, it treats language as a stable semantic anchor, depth observations as physical constraints, and temporal variation as a mechanism for resolving scale ambiguity. By integrating generative 3D shape priors as canonical geometric references, language-guided target selection, geometry-aware cross-modal fusion, and temporal consistency modeling, ProM-Pose constructs a metrically consistent and semantically grounded representation for previously unseen objects, improving robustness under partial observability and enhancing reliability in practical robotic and industrial environments. The main contributions of this work are summarized as follows:

• Category-Agnostic 9-DoF Perception via Generative Geometric Priors: We present a novel zero-shot framework that bypasses the need for predefined CAD libraries or category-specific templates. By leveraging a diffusion-based generative model, ProM-Pose synthesizes canonical geometric references directly from textual prompts. This strategy enables the recovery of anisotropic object scale and 6-DoF pose for previously unseen objects, effectively bridging the gap between open-vocabulary semantics and metric spatial understanding.
• Asymmetric Cross-Modal Attention (ACA) for Spatially Aware Fusion: To resolve the misalignment between linguistic anchors and geometric observations, we propose an ACA module. Unlike symmetric fusion, ACA treats dense visual tokens as queries to selectively aggregate multi-modal evidence from sparse geometric points and semantic text. This mechanism ensures that high-level semantic reasoning is grounded in local spatial correspondences, enhancing robustness against clutter and partial occlusions.
• Decoupled 9-DoF Reasoning with Temporal-Scale Observability: We introduce a decoupled decoding architecture with specialized anchors for rotation, translation, and scale to mitigate parameter coupling. By incorporating a shared temporal transformer and geometric consistency constraints across sequential frames, the framework explicitly improves scale observability under sparse-view and partial-observation conditions, achieving temporally stable and metrically consistent pose trajectories in unconstrained real-world environments.

2. Related Work

2.1. Geometry-Driven Pose Estimation

Early object pose estimation research primarily relied on explicit geometric modeling and known 3D object representations. Classic instance-level methods [7,8,9,10,11,12,13,45] typically learn correspondences between image observations and 3D model coordinates, followed by geometric refinement using PnP [46] or ICP [47]. These approaches achieve strong accuracy on standard benchmarks [45,48], but their dependence on instance-specific CAD models and dense annotations limits scalability in open-world environments.

To alleviate reliance on exact instance templates, category-level methods [14,15,16,17,18,19,49,50,51] learn canonical shape spaces shared across object categories, enabling joint pose and size estimation without per-instance CAD supervision. While these methods improve generalization within predefined categories, they remain constrained by category boundaries and typically handle scale through category-specific priors or normalization procedures [49]. Cross-category approaches such as Gen6D [33] further explore transferable representations, yet still exhibit instability under limited observations.

Overall, geometry-driven methods provide strong physical interpretability through explicit structural constraints, but their applicability to previously unseen objects and complex environments remains limited by prior model assumptions and observation completeness.

2.2. Generative-Prior and Model-Free Pose Estimation

To further reduce dependence on CAD models, recent work explores constructing object representations from sparse observations or generative priors. OnePose, OnePose++, and OVE6D [28,29,31,32,33] utilize sparse reference views or transferable feature embeddings to enable pose estimation of unseen objects. Although they improve flexibility, their performance remains sensitive to reference view quality and viewpoint sparsity.

Recent advances in generative 3D modeling have introduced automatically generated geometric priors. HIPPo [52] employs diffusion-based image-to-mesh generation combined with measurement-guided optimization, InstantPose [53] leverages large reconstruction models for single-view 3D shape recovery, and Any6D [54] jointly optimizes pose and size via render-and-compare strategies. These works demonstrate that generative shape priors can substitute manually created CAD models in zero-shot scenarios.

However, existing zero-shot solutions still face substantial challenges when tackling the full 9-DoF state. On the one hand, many model-free pipelines (e.g., FoundationPose [35]) improve robustness by enforcing cross-frame consistency, yet they typically assume known object size or isotropic scale, which limits their ability to handle unknown objects with complex aspect ratios. On the other hand, Diff9D [36] extends diffusion-based modeling to 9-DoF prediction, but category-level, learning pose distributions from closed-set training data remain, causing it to struggle to generalizing to open-vocabulary unseen categories. In contrast, ProM-Pose explicitly models anisotropic scale as a first-class variable. Unlike Diff9D, we leverage language-guided generative priors as a dynamic geometric reference to enable category-agnostic generalization. By integrating generative geometric priors, metric depth constraints, and temporal consistency modeling within a unified framework, ProM-Pose overcomes the anisotropic scale recovery limitations in methods such as FoundationPose, achieving more accurate and robust open-world 9-DoF pose estimation.

2.3. Language-Grounded and Open-Vocabulary Pose Estimation

Early language-conditioned perception methods [55,56,57,58,59] primarily operate at the 2D level, localizing target objects via bounding boxes or segmentation masks based on textual descriptions without explicit 3D geometry or scale modeling.

With the emergence of large-scale vision–language models [38,39], recent work explores language-guided spatial understanding. Text2Loc [40] and CityLoc [41] achieve text-driven camera localization at the scene level, while language-guided manipulation approaches [42] combine semantic grounding with category-level pose estimation. SoFar [60] further introduces semantic orientation prediction under open-vocabulary conditions. Nevertheless, these methods primarily focus on semantic reasoning, with limited modeling of complete object pose and anisotropic scale.

More recent open-vocabulary pose pipelines [43,44] combine language-based detection with downstream geometric alignment, enabling text-specified object selection. Open-world visual frameworks [61,62] further enhance language-based target retrieval. However, semantic information is typically applied at the global object level, while geometric registration is performed independently, resulting in limited semantic–geometric coupling for resolving local ambiguities under cluttered or occluded conditions.

Overall, existing research typically addresses object pose estimation from either geometric modeling, generative priors, or language grounding perspectives in isolation, lacking a unified framework that jointly integrates these complementary sources of information. Geometry-driven methods provide structural consistency but lack stable semantic guidance, generative approaches offer shape priors but often provide limited treatment of scale, and vision–language pipelines primarily emphasize target retrieval rather than metrically consistent 3D pose recovery. Consequently, achieving semantically controllable, geometrically consistent, and scale-aware zero-shot 9-DoF pose estimation under open-vocabulary conditions remains insufficiently explored. The proposed ProM-Pose framework addresses this gap by integrating language priors, generative 3D geometry, and depth- and temporal-based constraints within a unified formulation.

Overall, existing research typically addresses object pose estimation from either geometric modeling, generative priors, or language grounding perspectives in isolation, lacking a unified framework that jointly integrates these complementary sources of information for complex 9-DoF tasks. Geometry-driven methods provide structural consistency but lack stable semantic guidance, generative approaches offer shape priors but often provide limited treatment of scale, and vision–language pipelines primarily emphasize target retrieval rather than metrically consistent 3D pose recovery. Consequently, achieving semantically controllable, geometrically consistent, and scale-aware zero-shot 9-DoF pose estimation under open-vocabulary conditions remains insufficiently explored, especially due to the ambiguity between rotation and anisotropic scale. The proposed ProM-Pose framework addresses this gap by integrating language-conditioned generative 3D shape priors, depth- and temporal-based constraints, and a system-level design that bridges semantic grounding and metric geometric reasoning within a unified formulation.

3. Method

3.1. Overview

An overview of the proposed framework is illustrated in Figure 2. Given a continuous multi-modal observation sequence of length N, ${\mathcal{S}}_{1:N}={\left\{{\mathcal{S}}_t\right\}}_{t=1}^N,{\mathcal{S}}_t=\{{\mathcal{I}}_t,{\mathcal{D}}_t,\mathcal{T}\},$ where ${\mathcal{I}}_t\in {\mathbb{R}}^{H\times W\times 3}$ and ${\mathcal{D}}_t\in {\mathbb{R}}^{H\times W}$ denote the RGB image and the aligned depth map at time step t, respectively, $\mathcal{T}$ represents the textual semantic description of the target object O and remains fixed throughout the sequence, and the camera intrinsic matrix $\mathbf{K}$ is known. The object categories observed during training are strictly disjoint from the target object category at inference time. Under this open-set generalization setting, our goal is to estimate the nine degrees of freedom (9-DoF) pose of the target object with respect to the camera coordinate system over the entire sequence: ${\widehat{\mathcal{P}}}_{1:N}={\left\{{\widehat{\mathcal{P}}}_t\right\}}_{t=1}^N,{\widehat{\mathcal{P}}}_t=\{{\widehat{\mathbf{R}}}_t,{\widehat{\mathbf{T}}}_t,{\widehat{\mathbf{S}}}_t\},$ where ${\widehat{\mathbf{R}}}_t\in SO\left(3\right)$ denotes the rotation, ${\widehat{\mathbf{T}}}_t\in {\mathbb{R}}^3$ the translation, and ${\widehat{\mathbf{S}}}_t\in {\mathbb{R}}^3$ the anisotropic 3D scale. Our method does not learn instance- or category-specific pose priors. Instead, it aims to acquire category-agnostic geometric reasoning under cross-modal conditions. All vision–language encoders are employed with frozen pretrained weights to provide stable semantic and perceptual representations, while the cross-modal fusion and decoding modules are trained to achieve geometry-aware and category-independent pose estimation for previously unseen objects. The complete step-by-step algorithmic procedure of our framework is summarized in Algorithm 1.

Algorithm 1 ProM-Pose: Zero-Shot 9-DoF Object Pose Estimation

Require: RGB-D sequence ${\mathcal{S}}_{1:N}={\{{\mathcal{I}}_t,{\mathcal{D}}_t\}}_{t=1}^N$, Text prompt $\mathcal{T}$, Intrinsics K, Window size n

Ensure: 9-DoF poses ${\widehat{\mathcal{P}}}_{1:N}={\{{\widehat{R}}_t,{\widehat{T}}_t,{\widehat{S}}_t\}}_{t=1}^N$

1: $T_{text}\leftarrow {\varphi }_T\left(\mathcal{T}\right)$   2: ${\overline{P}}_O\leftarrow \mathrm{GeneratePrior}({\mathcal{I}}_1,\mathcal{T})$   3: $G_O^{can}\leftarrow {\varphi }_G\left({\overline{P}}_O\right)$                    ▷ Extract canonical prior geometry   4: for $t=1$ to N do   5:       $V_t\leftarrow {\varphi }_V\left({\mathcal{I}}_t^O\right)$                        ▷ Visual token extraction   6:       $P_t^O\leftarrow \mathrm{BackProject}({\mathcal{D}}_t^O,K)$   7:       $G_t^{met}\leftarrow {\varphi }_G\left(P_t^O\right)$                     ▷ Metric geometric encoding   8:       $G_t^{can}\leftarrow {\varphi }_G\left(\mathrm{Normalize}\left(P_t^O\right)\right)$              ▷ Canonical geometric encoding   9:       $C_t\leftarrow {\mathrm{Proj}}_G\left(G_t^{can}\right)\oplus {\mathrm{Proj}}_G\left(G_O^{can}\right)\oplus {\mathrm{Proj}}_T\left(T_{text}\right)$ 10:       $F_t\leftarrow \mathrm{ACA}(Q=V_t,K=C_t,V=C_t)$        ▷ Asymmetric Cross-Modal Fusion 11:       $\{Z_t^R,Z_t^T,Z_t^S\}\leftarrow \mathrm{PoseTokenExtract}(F_t,P_t^O,G_t^{met})$ 12:       $Z_t\leftarrow Z_t^R\oplus Z_t^T\oplus Z_t^S$ 13: end for 14: for $t=1$ to N do 15:      $H_{1:n}\leftarrow {\varphi }_{temp}(Z_{1:n}+P_{1:n})$          ▷ Temporal modeling over sliding window 16:      $\{{\widehat{R}}_t,{\widehat{T}}_t\}\leftarrow {\mathrm{MLP}}_{RT}\left(\mathrm{CrossAttn}(H_{1:n},G_O^{can})\right)$        ▷ Structure-aligned decoding 17:      ${\widehat{S}}_t\leftarrow {\mathrm{MLP}}_S\left(H_{1:n}\right)$            ▷ Metric-preserved anisotropic scale decoding 18: end for 19: return ${\widehat{\mathcal{P}}}_{1:N}={\{{\widehat{R}}_t,{\widehat{T}}_t,{\widehat{S}}_t\}}_{t=1}^N$

3.2. Visual and Text Encoding

We first employ the open-vocabulary object detector Grounded-SAM 2 [61,63,64] with the textual prompts $\mathcal{T}$ to localize the target region. The detector outputs a 2D bounding box and a segmentation mask, which we use to crop the input into an object-centric RGB image ${\mathcal{I}}_t^O$ and a clean depth map ${\mathcal{D}}_t^O$. This effectively suppresses background clutter, focusing the model on appearance and geometric cues relevant to the object.

We adopt DINOv2 [65] as the visual encoder ${\varphi }_V$ to extract multi-scale spatial features, ${\mathbf{V}}_t={\varphi }_V\left({\mathcal{I}}_t^O\right).$ Meanwhile, BERT [66] serves as the text encoder ${\varphi }_T$ to encode $\mathcal{T}$ into token-level semantic representations, ${\mathbf{T}}_{\mathrm{text}}={\varphi }_{T_{\mathrm{text}}}\left(\mathcal{T}\right).$ Unlike CLIP-style methods [38,62] that align global embeddings, this design emphasizes spatial awareness and local correspondence, providing a stable semantic anchor for subsequent fusion.

3.3. Depth and Geometry Encoding

Given camera intrinsic parameters $\mathbf{K}$, we back-project ${\mathcal{D}}_t^O$ into a metric 3D point cloud ${\mathbf{P}}_t^O$ to provide physical grounding:

$${\mathbf{P}}_t^O=\left\{{\mathbf{X}}_t(u,v)={\mathcal{D}}_t^O(u,v){\mathbf{K}}^{-1}{[u,v,1]}^{\top }\right\}$$

(1)

where $(u,v)$ denotes the pixel coordinate.

To disentangle rotation reasoning from metric-dependent scale and translation, we introduce a dual geometric encoding strategy. Specifically, metric geometry ${\mathbf{P}}_t^O$ preserves absolute spatial measurements essential for translation and scale estimation, while canonical geometry provides viewpoint-invariant shape cues that stabilize rotation prediction under varying object distances and sizes.

To obtain a shape-oriented representation stable for rotation reasoning, we construct a canonicalized version of the observed point cloud ${\overline{\mathbf{P}}}_t^O$ by centering ${\mathbf{P}}_t^O$ and normalizing it into a unit axis-aligned bounding box. Additionally, we use the diffusion-based generator Hunyuan3D [67] to produce an object-level mesh ${\mathcal{M}}_O$ conditioned on $\mathcal{T}$ and ${\mathcal{I}}_t^O$. The mesh is generated once per sequence. The generated mesh is used purely as a structural canonical reference and does not provide metric scale or instance-specific geometry supervision. We sample a point cloud from the generated mesh and similarly normalize it into a canonical prior point cloud ${\overline{\mathbf{P}}}_O$.

We then adopt Point Transformer [68] as a shared geometry encoder ${\varphi }_G$ to extract three complementary feature streams:

$${\mathbf{G}}_t^{\mathrm{met}}={\varphi }_G\left({\mathbf{P}}_t^O\right),{\mathbf{G}}_t^{\mathrm{can}}={\varphi }_G\left({\overline{\mathbf{P}}}_t^O\right),{\mathbf{G}}_O^{\mathrm{can}}={\varphi }_G\left({\overline{\mathbf{P}}}_O\right)$$

(2)

where ${\mathbf{P}}_t^O$ denotes the observed metric point cloud, ${\overline{\mathbf{P}}}_t^O$ is its canonicalized counterpart, and ${\overline{\mathbf{P}}}_O$ represents the generated canonical prior. The encoder ${\varphi }_G$ is weight-shared across all three inputs.

This decoupled encoding explicitly separates metric grounding (${\mathbf{G}}_t^{\mathrm{met}}$) from shape-centric reasoning (${\mathbf{G}}_t^{\mathrm{can}},{\mathbf{G}}_O^{\mathrm{can}}$).

3.4. Asymmetric Cross-Modal Attention Mechanism

To construct semantic-aware and geometrically consistent representations, we introduce an asymmetric cross-modal attention (ACA) module as shown in Figure 3. This module treats visual tokens ${\mathbf{V}}_t$ as queries and uses textual and geometric features as keys and values. By anchoring the attention on visual tokens, the model selectively retrieves semantic and geometric evidence at spatial locations corresponding to the observed object, avoiding unordered feature mixing.

For alignment, we use canonical geometry tokens (${\mathbf{G}}_t^{\mathrm{can}},{\mathbf{G}}_O^{\mathrm{can}}$) as they provide stable shape cues invariant to viewpoint-dependent scale and translation. The fused cross-modal context ${\mathbf{C}}_t$ and the multi-modal representation ${\mathbf{F}}_t$ are computed as follows:

$${\mathbf{C}}_t={\mathrm{Proj}}_G\left({\mathbf{G}}_t^{\mathrm{can}}\right)\oplus {\mathrm{Proj}}_G\left({\mathbf{G}}_O^{\mathrm{can}}\right)\oplus {\mathrm{Proj}}_T\left({\mathbf{T}}_{\mathrm{text}}\right),$$

(3)

$${\mathbf{F}}_t=\mathrm{softmax}\left({\displaystyle \frac{\left({\mathbf{V}}_t{\mathbf{W}}_Q\right){\left({\mathbf{C}}_t{\mathbf{W}}_K\right)}^{\top }}{\sqrt{d_k}}}\right)\left({\mathbf{C}}_t{\mathbf{W}}_V\right).$$

(4)

where ${\mathrm{Proj}}_G(·)$ and ${\mathrm{Proj}}_T(·)$ denote learnable projection layers; ⊕ represents feature concatenation; ${\mathbf{W}}_Q$, ${\mathbf{W}}_K$, and ${\mathbf{W}}_V$ denote learnable query, key, and value projection matrices; and $d_k$ denotes the dimensionality of the key and query vectors in the attention mechanism.

3.5. Pose Token Extraction

To extract compact pose-relevant representations from ${\mathbf{F}}_t$, we employ a query-based aggregation strategy as illustrated in Figure 4.

We first derive a geometry-informed initialization by extracting a lightweight descriptor ${\mathbf{d}}_t$ from ${\mathbf{P}}_t^O$. We suppress depth outliers within the $[p_5,p_{95}]$ percentile range of the depth distribution and compute the centroid ${\mathbf{c}}_t$, principal directions ${\mathbf{U}}_t$ via Principal Component Analysis (PCA) applied to the covariance ${\Sigma }_t$, and coarse object extents ${\mathbf{e}}_t$ defined by the axis-aligned bounding box:

$${\mathbf{c}}_t={\displaystyle \frac{1}{|{\mathbf{P}}_t^O|}}\sum _{\mathbf{x}\in {\mathbf{P}}_t^O}\mathbf{x},{\Sigma }_t={\displaystyle \frac{1}{|{\mathbf{P}}_t^O|}}\sum _{\mathbf{x}\in {\mathbf{P}}_t^O}(\mathbf{x}-{\mathbf{c}}_t){(\mathbf{x}-{\mathbf{c}}_t)}^{\top },$$

(5)

$${\mathbf{d}}_t={\mathbf{c}}_t\oplus \mathrm{vec}\left({\mathbf{U}}_t\right)\oplus {\mathbf{e}}_t.$$

(6)

where ${\mathbf{U}}_t$ denotes the PCA principal directions, and $\mathrm{vec}(·)$ represents vectorization.

To preserve fine-grained metric geometry, we aggregate ${\mathbf{G}}_t^{\mathrm{met}}$ via global mean pooling to obtain ${\mathbf{g}}_t^{\mathrm{met}}$. We then define two specialized descriptors: ${\mathbf{d}}_t^R={\mathbf{d}}_t$ for rotation to avoid metric bias, and ${\mathbf{d}}_t^{TS}={\mathbf{d}}_t\oplus {\mathbf{g}}_t^{\mathrm{met}}$ for translation and scale. These descriptors modulate learnable query embeddings ${\mathbf{Q}}_{*}^{\left(0\right)}$ to produce geometry-conditioned pose queries ${\tilde{\mathbf{Q}}}_{*}$. Here, ${\mathbf{Q}}_{*}^{\left(0\right)}$ denotes learnable query embeddings and ${\mathbf{g}}_t^{\mathrm{met}}$ is obtained via global mean pooling.

The final pose tokens $\{{\mathbf{Z}}_t^R,{\mathbf{Z}}_t^T,{\mathbf{Z}}_t^S\}$ are extracted via cross-attention between ${\tilde{\mathbf{Q}}}_{*}$ and ${\mathbf{F}}_t$.

$${\tilde{\mathbf{Q}}}_R={\mathbf{Q}}_R^{\left(0\right)}+{\mathrm{MLP}}_R\left({\mathbf{d}}_t^R\right),{\tilde{\mathbf{Q}}}_T={\mathbf{Q}}_T^{\left(0\right)}+{\mathrm{MLP}}_T\left({\mathbf{d}}_t^{TS}\right),{\tilde{\mathbf{Q}}}_S={\mathbf{Q}}_S^{\left(0\right)}+{\mathrm{MLP}}_s\left({\mathbf{d}}_t^{TS}\right).$$

(7)

$${\mathbf{Z}}_t^R=\mathrm{Attn}({\tilde{\mathbf{Q}}}_R,{\mathbf{F}}_t,{\mathbf{F}}_t),{\mathbf{Z}}_t^T=\mathrm{Attn}({\tilde{\mathbf{Q}}}_T,{\mathbf{F}}_t,{\mathbf{F}}_t),{\mathbf{Z}}_t^S=\mathrm{Attn}({\tilde{\mathbf{Q}}}_S,{\mathbf{F}}_t,{\mathbf{F}}_t).$$

(8)

where $\mathrm{Attn}(·)$ denotes a standard cross-attention operator.

3.6. Decoupled Pose Decoding and Temporal Refinement

To resolve the intrinsic coupling between orientation and scale, we implement a metric–canonical decoupling strategy that disentangles pose components based on their geometric dependencies. Specifically, rotation tokens ${\mathbf{Z}}_t^R$ are grounded in viewpoint-invariant canonical cues $({\mathbf{G}}_t^{can},{\mathbf{G}}_O^{can})$, while translation and scale tokens $({\mathbf{Z}}_t^T,{\mathbf{Z}}_t^S)$ leverage absolute metric measurements ${\mathbf{G}}_t^{met}$ to ensure physical consistency. To further mitigate frame-wise jitter and partial observability, these specialized pose tokens are aggregated and processed through a shared temporal transformer ${\varphi }_{\mathrm{temp}}$ to model cross-frame geometric consistency (Figure 5).

For each frame t, we aggregate the pose tokens into a per-frame representation ${\mathbf{Z}}_t={\mathbf{Z}}_t^R\oplus {\mathbf{Z}}_t^T\oplus {\mathbf{Z}}_t^S$. Given a sliding temporal window of n frames, we incorporate learnable positional encodings ${\mathbf{P}}_{1:n}$ to preserve temporal ordering, obtaining refined features:

$${\mathbf{H}}_{1:n}={\varphi }_{\mathrm{temp}}({\mathbf{Z}}_{1:n}+{\mathbf{P}}_{1:n}),$$

(9)

where ${\mathbf{H}}_{1:n}$ serves as a consistent geometric representation shared by all downstream tasks.

We disentangle the 9-DoF estimation into two specialized heads. The Rotation–Translation Head conditions the shared features ${\mathbf{H}}_{1:N}$ on canonical prior shape tokens $\mathbf{S}$ (derived from ${\mathbf{G}}_O^{\mathrm{can}}$) to stabilize orientation alignment. This ensures that the estimated rotation and translation are grounded in stable structural references:

$$\{{\widehat{\mathbf{R}}}_t,{\widehat{\mathbf{T}}}_t\}={\mathrm{MLP}}_{RT}\left({\varphi }_{RT}^{\mathrm{head}}({\mathbf{H}}_{1:N},\mathbf{S})\right).$$

(10)

In contrast, the Anisotropic Scale Head operates solely on the shared temporal features ${\mathbf{H}}_{1:N}$ without prior conditioning. This ensures that the scale ${\widehat{\mathbf{S}}}_t$ is inferred exclusively from metric observations to avoid biases from the generative prior:

$${\widehat{\mathbf{S}}}_t={\mathrm{MLP}}_S\left({\varphi }_S^{\mathrm{head}}\left({\mathbf{H}}_{1:N}\right)\right).$$

(11)

This task-specific decoupling enables our framework to balance stable structural alignment with robust metric grounding for zero-shot 9-DoF pose estimation.

3.7. Optimization Objectives and Loss Functions

Our method is trained end-to-end by jointly optimizing pose regression, temporal smoothness, and geometric consistency losses. The overall objective is

$$\mathcal{L}={\mathcal{L}}_{\mathrm{pose}}+{\lambda }_{\mathrm{temp}}{\mathcal{L}}_{\mathrm{temp}}+{\lambda }_{\mathrm{geo}}{\mathcal{L}}_{\mathrm{geo}},$$

(12)

where ${\lambda }_{\mathrm{temp}}$ and ${\lambda }_{\mathrm{geo}}$ balance the corresponding terms.

We supervise the 9-DoF pose predictions over the full sequence:

$${\mathcal{L}}_R^{\left(t\right)}=\parallel log\left({\widehat{\mathbf{R}}}_t^{\top }{\mathbf{R}}_t\right){\parallel }_2,{\mathcal{L}}_T^{\left(t\right)}=\parallel {\widehat{\mathbf{T}}}_t-{\mathbf{T}}_t{\parallel }_1,{\mathcal{L}}_S^{\left(t\right)}={\parallel {\widehat{\mathbf{S}}}_t-{\mathbf{S}}_t\parallel }_1.$$

(13)

$${\mathcal{L}}_{\mathrm{pose}}={\displaystyle \frac{1}{N}}\sum _{t=1}^N\left({\mathcal{L}}_R^{\left(t\right)}+{\mathcal{L}}_T^{\left(t\right)}+{\mathcal{L}}_S^{\left(t\right)}\right).$$

(14)

To suppress frame-wise jitter under occlusions and noisy observations, we penalize abrupt pose changes between adjacent frames:

$${\mathcal{L}}_{\mathrm{temp}}=\sum _{t=2}^N\left(\parallel {\widehat{\mathbf{T}}}_t-{\widehat{\mathbf{T}}}_{t-1}{\parallel }_2^2+\parallel log\left({\widehat{\mathbf{R}}}_{t-1}^{\top }{\widehat{\mathbf{R}}}_t\right){\parallel }_2^2+{\parallel {\widehat{\mathbf{S}}}_t-{\widehat{\mathbf{S}}}_{t-1}\parallel }_2^2\right).$$

(15)

Geometric consistency loss is the key constraint for resolving scale ambiguity by enforcing alignment between the pose-transformed canonical shape prior and the metric depth observations in 3D space. We first transform the canonical prior point cloud ${\overline{\mathbf{P}}}_O$ into the camera coordinate system. We then measure the discrepancy between ${\widehat{\mathbf{P}}}_t^c$ and the mask-filtered observation point cloud ${\mathbf{P}}_t^O$ using the bidirectional Chamfer Distance.

$${\widehat{\mathbf{P}}}_t^c=\left\{{\widehat{\mathbf{R}}}_t\mathrm{diag}\left({\widehat{\mathbf{S}}}_t\right){\overline{\mathbf{x}}}^o+{\widehat{\mathbf{T}}}_t|{\overline{\mathbf{x}}}^o\in {\overline{\mathbf{P}}}_O\right\}.$$

(16)

$${\mathcal{L}}_{\mathrm{geo}}={\displaystyle \frac{1}{N}}\sum _{t=1}^N\left({\displaystyle \frac{1}{|{\widehat{\mathbf{P}}}_t^c|}}\sum _{\mathbf{x}\in {\widehat{\mathbf{P}}}_t^c}\underset{\mathbf{y}\in {\mathbf{P}}_t^O}{min}{\parallel \mathbf{x}-\mathbf{y}\parallel }_2^2+{\displaystyle \frac{1}{|{\mathbf{P}}_t^O|}}\sum _{\mathbf{y}\in {\mathbf{P}}_t^O}\underset{\mathbf{x}\in {\widehat{\mathbf{P}}}_t^c}{min}{\parallel \mathbf{y}-\mathbf{x}\parallel }_2^2\right).$$

(17)

4. Experiments

4.1. Implementation Details

For open-vocabulary object detection and segmentation, we adopt the Grounded-SAM framework [61], integrating Grounding DINO 1.5 [63] for text-conditioned object localization and SAM 2 [64] for mask generation in image sequences. Specifically, we use the official pretrained Grounding DINO 1.5 (Swin-T backbone) checkpoint for bounding box prediction and the SAM 2 Hiera-L model for video-aware segmentation. Both models are used in a frozen manner without task-specific fine-tuning. The resulting text-guided bounding boxes and temporally consistent object masks are utilized to crop object-centric RGB-D observations for downstream processing. For the visual backbone, we employ the frozen DINOv2 (ViT-g/14) [65] pretrained weights, extracting features from the last layer with a spatial resolution of $37\times 37$ (for a $518\times 518$ input). The textual description is encoded into a 768-dimensional sequence of tokens by a BERT-Base encoder [66]. The Geometry Encoder ${\varphi }_G$ is based on Point Transformer v2 [68] with a hidden dimension of 256. Both the asymmetric cross-modal attention (ACA) module and the Temporal Transformer ${\varphi }_{\mathrm{temp}}$ utilize a multi-head mechanism with 8 heads and a latent dimension of 512, which follows the standard transformer configuration widely adopted in cross-modal perception and vision transformers [69], providing sufficient representation capacity while maintaining computational efficiency. For the temporal modeling, we set the sliding window length to $n=16$, which provides a balance between long-term consistency and computational efficiency. We evaluated temporal window lengths within the range of 8–24 frames. Shorter windows weaken multi-view geometric consistency and increase frame-wise jitter, whereas longer windows introduce additional memory overhead with only marginal accuracy improvement. Based on this analysis, we select $n=16$ as it provides a favorable trade-off between temporal robustness and computational efficiency.

The model is trained end-to-end for 20 epochs using the Adam optimizer [70] with ${\beta }_1=0.9, {\beta }_2=0.999$. We adopt a cosine annealing scheduler [71] with an initial learning rate of $1\times {10}^{-4}$, decaying to $1\times {10}^{-5}$ at a weight decay of $5\times {10}^{-4}$. To enhance the robustness of category-agnostic estimation, we apply online data augmentation, including random color jittering for RGB images and random point dropout (up to 20%) for depth-projected point clouds. In addition, we evaluated a grid of loss–weight combinations with ${\lambda }_{\mathrm{geo}}\in \{1,3,5,10,15\}$ and ${\lambda }_{\mathrm{temp}}\in \{0.1,0.3,0.5,0.7,1.0\}$. We observe that smaller geometric weights lead to insufficient structural constraints and unstable anisotropic scale estimation, whereas excessively large values overly restrict optimization and slow convergence. The temporal weight controls the trade-off between jitter suppression and motion responsiveness: small values fail to enforce temporal consistency, while large values introduce motion over-smoothing. Based on these observations, the configuration $({\lambda }_{\mathrm{geo}},{\lambda }_{\mathrm{temp}})=(10.0,0.5)$ is selected as it achieves the best balance between geometric accuracy and temporal stability. We further observe stable performance within a reasonable neighborhood around this configuration, indicating that the proposed framework does not rely on carefully tuned loss weighting.

For each sequence, we generate a canonical mesh via Hunyuan3D-Shape-v2-1 [67] using the first frame’s RGB image and textual prompts. We uniformly sample $N_p=5000$ points from the mesh surface to form the canonical prior ${\overline{\mathbf{P}}}_O$. The observed point cloud ${\mathbf{P}}_t^O$ is derived from the depth map using known intrinsic $\mathbf{K}$ and is downsampled to 2048 points using Farthest Point Sampling (FPS) to ensure uniform spatial coverage. During the PCA-based descriptor extraction, we utilize a robust covariance estimation by clipping depth outliers within the $[5\%,95\%]$ percentile range.

All experiments are conducted on a single NVIDIA L40 GPU (48GB RAM) (NVIDIA Corp., Santa Clara, CA, USA). The training process takes approximately 10 h for the full dataset. At inference time, our framework achieves a frame rate of 12 FPS. Note that the mesh generation by Hunyuan3D [67] is performed only once per sequence, which does not bottleneck the continuous tracking phase.

4.2. Datasets

We evaluate ProM-Pose on two public category-level RGB-D benchmarks, CAMERA25 [49], and REAL275 [49], as well as a self-collected real-world dataset LogiScan.

CAMERA25 [49] is a widely used large-scale synthetic RGB-D dataset featuring six object categories with instance-level 9-DoF annotations. We construct the textual prompts $\mathcal{T}$ by using basic category names (e.g., ‘bottle’, ‘can’, ‘camera’) as primary anchors. Following the standard protocol, we use only the CAMERA25 training/validation sets during training and evaluate exclusively on the CAMERA25 test set to ensure no overlap between training and testing instances.

REAL275 [49] is a real-world RGB-D dataset complementary to CAMERA25, covering the same six object categories to evaluate cross-instance generalization in real environments. Similar to the synthetic evaluation, we use category names as primary anchors for quantitative benchmarking. During inference, the framework further demonstrates its open-vocabulary flexibility by processing descriptive phrases such as ‘brown open laptop’. In our zero-shot setting, we do not train on REAL275 and evaluate exclusively on the officially provided 2754 test images.

LogiScan is a real-world RGB-D dataset collected in logistics and warehouse scenarios using an Azure Kinect sensor (Microsoft Corp., Redmond, WA, USA) as shown in Figure 6. It includes pallets, cardboard boxes, and irregularly shaped packages with various stacking patterns, occlusions, and lighting variations. We perform depth-to-color alignment and center-field-of-view cropping on the raw RGB-D sequences and update the camera intrinsics accordingly to ensure pixel-wise correspondence and geometric consistency [72]. For this dataset, we construct the textual prompts $\mathcal{T}$ using detailed descriptive phrases (e.g., ‘Yellow Wooden Pallet’ and ‘Industrial Wooden Pallet’). Since obtaining accurate 9-DoF ground-truth labels for large or heavily occluded objects in this setting is impractical, we mainly use LogiScan for qualitative evaluation to assess robustness, temporal stability, and scale consistency in industrial scenarios.

4.3. Evaluation Metrics

Following the widely adopted evaluation scheme [14,15,36,49], to quantitatively evaluate pose estimation performance, we employ two types of metrics: 3D Intersection-over-Union (IoU3D) and $n^{°},m\mathrm{cm}$. Specifically, the mean Average Precision (mAP%) of IoU3D and $n^{°},m\mathrm{cm}$ threshold are reported as the final evaluation results.

We compute the IoU3D between the predicted 9-DoF oriented bounding box and the ground-truth bounding box:

$$\mathrm{IoU}3\mathrm{D}={\displaystyle \frac{V(B_{\mathrm{pred}}\cap B_{\mathrm{gt}})}{V(B_{\mathrm{pred}}\cup B_{\mathrm{gt}})}},$$

(18)

where $B_{\mathrm{pred}}$ and $B_{\mathrm{gt}}$ denote the predicted and ground-truth 3D bounding boxes, respectively, $V(·)$ denotes volume, and ∩ and ∪ denote intersection and union. A sample is considered correct when IoU3D exceeds a given threshold. We report accuracy at thresholds of 0.5 and 0.75, denoted as 3D₅₀ and 3D₇₅, respectively.

The $n^{°},m\mathrm{cm}$ metric considers a predicted pose correct only if the rotation error $\Delta R$ and translation error $\Delta T$ are within $n^{°}$ and $m\mathrm{cm}$, respectively. Given the predicted pose $({\widehat{\mathbf{R}}}_t,{\widehat{\mathbf{T}}}_t)$ and the ground-truth pose $({\mathbf{R}}_t,{\mathbf{T}}_t)$, the errors and the overall accuracy are defined as

$$\Delta R_t=arccos\left({\displaystyle \frac{\mathrm{Tr}\left({\widehat{\mathbf{R}}}_t{\mathbf{R}}_t^T\right)-1}{2}}\right),\Delta T_t={\parallel {\widehat{\mathbf{T}}}_t-{\mathbf{T}}_t\parallel }_2$$

(19)

$$\mathrm{Accuracy}={\displaystyle \frac{1}{N}}\sum _{t=1}^N⊮(\Delta R_t<n^{°}\wedge \Delta T_t<m\mathrm{cm})$$

(20)

where $arccos(·)$ denotes the inverse cosine function used to compute the angular distance; $\mathrm{Tr}(·)$ is the trace operator, representing the sum of the diagonal elements of a matrix; ${\parallel ·\parallel }_2$ denotes the Euclidean distance (L2 norm) between two vectors; cm is the unit for translation error; N is the total number of samples; and $⊮(·)$ is the indicator function that outputs 1 if the condition is satisfied and 0 otherwise. Similarly to most of the previous works, we use $5^{°},2\mathrm{cm}$, $5^{°},5\mathrm{cm}$, ${10}^{°},2\mathrm{cm}$, and ${10}^{°},5\mathrm{cm}$ for evaluation.

4.4. Quantitative Results

We compare ProM-Pose with several representative approaches, including shape-prior-based methods SPD [15] and SGPA [50], the transformer-guided shape reconstruction method STG6D [51], the 9-DoF diffusion-based model Diff9D [36], and the open-vocabulary 6D pose-estimation method Horyon [44]. For training on CAMERA25, all methods use synthetic data only. For REAL275, some baselines use a mixed-data strategy by randomly selecting images from REAL275 and CAMERA25 at a ratio of 1:3 in each batch. Diff9D, Horyon, and our ProM-Pose are evaluated under a strict zero-shot setting, training exclusively on synthetic data to demonstrate its generalization capability. To ensure statistical robustness, all REAL275 results for ProM-Pose are averaged over three independent runs with different random seeds, and we report the mean and standard deviation. Quantitative results are summarized in

Table 1. Quantitative comparison on the CAMERA25 and REAL275 datasets. Performance is evaluated using mean Average Precision (mAP%) across different 3D IoU thresholds and $n^{°}m$ cm pose error metrics. Our ProM-Pose achieves state-of-the-art results under the zero-shot setting, demonstrating superior generalization compared to existing methods. Bold values indicate the best performance.

Data	Method	Training Data	mean Average Precision (mAP%)
			3D50	3D75	5°2 cm	5°5 cm	10°2 cm	10°5 cm
CAMERA25	SPD [15]	Syn. Only	93.2	83.1	54.3	59.0	73.3	78.6
	SGPA [50]	Syn. Only	93.2	88.1	70.7	74.5	82.7	88.4
	STG6D [51]	Syn. Only	93.1	89.1	73.0	77.0	84.5	89.8
	Diff9D [36]	Syn. Only	93.4	86.2	77.9	72.5	82.0	87.5
	Horyon [44]	Syn. Only	93.1	88.5	71.5	75.5	83.5	88.8
	ProM-Pose (Ours)	Syn. Only	93.6	90.0	75.0	78.5	84.3	90.5
REAL275	SPD [15]	Mixed (Syn.+Real)	77.3	53.2	19.3	21.4	43.2	54.1
	SGPA [50]	Mixed (Syn.+Real)	80.1	61.9	35.9	39.6	61.3	70.7
	STG6D [51]	Mixed (Syn.+Real)	81.6	65.8	41.6	46.1	63.8	73.9
	Diff9D [36]	Zero-shot (Syn.)	76.5	41.7	35.3	43.9	54.8	70.0
	Horyon [44]	Zero-shot (Syn.)	77.0	43.2	37.9	40.6	58.1	72.0
	ProM-Pose (Ours)	Zero-shot (Syn.)	$77.8\pm 0.15$	$50.9\pm 0.29$	$42.2\pm 0.37$	$49.0\pm 0.56$	$60.5\pm 0.18$	$76.0\pm 0.23$

.

On the synthetic CAMERA25 benchmark, ProM-Pose achieves state-of-the-art or competitive performance across both 3D IoU and strict pose metrics. In particular, our method demonstrates robust estimation capabilities on $Io{U}_{50}$ and $Io{U}_{75}$ while consistently outperforming shape-prior baselines SPD and SGPA under high-precision thresholds such as $5^{°}5 \mathrm{cm}$ and ${10}^{°}5 \mathrm{cm}$. Notably, ProM-Pose remains highly competitive with the temporal-based STG6D, despite the latter’s reliance on sequential modeling. These results validate that integrating text-conditioned generative 3D shape priors with metric depth and cross-modal geometric reasoning enables accurate 9-DoF pose and scale estimation without requiring category-level CAD models. The lower performance of Diff9D on this benchmark suggests that its diffusion sampling strategy may not leveraging the simplified data distribution of synthetic environments as effectively as our direct geometric approach.

REAL275 is a significantly more challenging real-world dataset designed to evaluate cross-domain generalization. Following a rigorous zero-shot protocol, our model is trained exclusively on CAMERA25 and evaluated directly on REAL275. ProM-Pose achieves the best or top-tier performance across IoU and strict pose metrics, notably outperforming SPD, SGPA, and STG6D under $Io{U}_{75}$ and $5^{°}2 \mathrm{cm}$. Critically, while SPD and SGPA benefit from mixed CAMERA25+REAL275 training data, our model surpasses them using only synthetic supervision. This indicates that the synergy between language-conditioned generative priors and temporal geometric consistency effectively mitigates the sim-to-real domain gap and enhances robustness against occlusion, sensor noise, and background clutter. While Diff9D shows marginal gains under relaxed thresholds due to its stochastic hypothesis exploration, ProM-Pose exhibits a clear advantage under stricter criteria, demonstrating more precise rotation, translation, and anisotropic scale recovery. Compared with the open-vocabulary method Horyon, which primarily relies on 2D vision–language alignment, ProM-Pose achieves substantially higher accuracy on strict geometric metrics, highlighting the indispensable role of explicit 3D geometric modeling for fine-grained category-level pose estimation.

4.5. Qualitative Evaluation

4.5.1. Results on Public Benchmarks

Figure 7 presents qualitative pose estimation results on CAMERA25 and REAL275. Despite cluttered backgrounds and partial occlusions, ProM-Pose produces tightly aligned 3D bounding boxes, demonstrating stable geometric alignment and consistent pose estimation across viewpoints.

Figure 8 shows representative open-vocabulary 9-DoF pose estimation results on REAL275 conditioned on textual prompts. Even under appearance variations and occlusions, ProM-Pose consistently localizes the target object and recovers geometrically consistent poses, validating the effectiveness of language-conditioned generative priors and cross-modal fusion.

To further analyze robustness under varying tolerances, Figure 9 reports average precision curves over IoU, rotation, and translation thresholds. ProM-Pose consistently maintains higher precision than SGPA on REAL275, particularly under stricter criteria, indicating improved geometric alignment and stronger cross-domain generalization.

4.5.2. Industrial Real-World Validation

We further evaluate ProM-Pose on the self-collected LogiScan dataset captured in logistics and warehouse environments using an Azure Kinect sensor. As shown in Figure 10, under normal illumination, the model maintains consistent 3D alignment across consecutive frames.

Under challenging lighting conditions (Figure 11), ProM-Pose remains robust across most frames. Failure occurs only when severe illumination degradation and limited visibility reduce both visual and depth reliability.

4.6. Inference Efficiency

To ensure a fair comparison, we evaluate ProM-Pose and Diff9D on the same hardware platform using a single NVIDIA L40 GPU under the REAL275 evaluation setting. ProM-Pose achieves approximately 12 FPS, while Diff9D runs at around 18 FPS using the official implementation and default inference configuration. Despite the lower frame rate, ProM-Pose achieves substantially higher accuracy under strict pose metrics while producing stable predictions through a deterministic single forward pass. In contrast to diffusion-based iterative inference, the proposed method maintains consistent latency and avoids sampling-induced variability. Future improvements may incorporate efficient attention mechanisms such as FlashAttention [73].

4.7. Ablation Study

We conduct systematic ablation experiments on the REAL275 dataset to analyze the contribution of each component in ProM-Pose. Ablation results are reported from a single representative run (seed closest to the mean performance), while the main REAL275 results are averaged over three runs (mean ± std). Quantitative results are summarized in

Table 2. Ablation results on REAL275. Gen3D: Generative 3D Prior, ACA: Asymmetric Cross-Modal Attention Module, MCD: Metric–Canonical Decoupling Strategy, TempC: Temporal Consistency Module, ${\mathcal{L}}_{\mathrm{geo}}$: Geometric Consistency Loss. ✓ indicates that the corresponding component is utilized.

Variant	Gen3D	ACA	MCD	TempC	${\mathcal{L}}_{\mathbf{geo}}$	3D50	3D75	5°2 cm	10°5 cm
(1) Full Model	✓	✓	✓	✓	✓	77.92	50.78	42.50	76.12
(2) w/o Gen3D		✓	✓	✓	✓	75.10	44.95	38.60	71.40
(3) w/o ACA	✓		✓	✓	✓	76.55	48.25	41.05	74.70
(4) w/o MCD	✓	✓		✓	✓	76.05	47.10	39.75	73.35
(5) w/o TempC	✓	✓	✓		✓	76.25	47.45	40.05	73.05
(6) w/o ${\mathcal{L}}_{\mathrm{geo}}$	✓	✓	✓	✓		76.70	47.85	41.30	74.25

, which presents the ablation study on REAL275 to examine the contribution of each module in ProM-Pose.

Removing the generative 3D prior (Gen3D) leads to the most significant degradation under strict pose thresholds, particularly for the $5^{°},2\mathrm{cm}$ metric. This observation indicates that the language-conditioned canonical prior provides essential structural guidance for unseen objects. Without this prior, rotation estimation becomes more vulnerable to partial observations and category-level ambiguities.

The metric–canonical decoupling strategy (MCD) and temporal consistency modeling (TempC) also contribute substantially. Eliminating MCD increases translation and anisotropic scale errors, confirming that separating metric-dependent reasoning from canonical shape representation mitigates optimization conflicts. Similarly, removing the temporal module results in consistent performance decline across all evaluation metrics, suggesting that multi-frame aggregation effectively suppresses frame-wise jitter and enhances robustness under occlusion and noisy depth measurements.

The asymmetric cross-modal attention (ACA) and the geometric consistency loss ${\mathcal{L}}_{\mathrm{geo}}$ further provide complementary improvements. Replacing ACA with symmetric fusion weakens semantic–geometric alignment, highlighting the benefit of spatially anchored cross-modal interaction. Meanwhile, eliminating ${\mathcal{L}}_{\mathrm{geo}}$ particularly affects IoU₇₅, demonstrating its role in enforcing fine-grained 3D alignment and stabilizing anisotropic scale estimation.

Overall, each module contributes to a distinct aspect of the 9-DoF estimation process, including semantic grounding, metric-scale disentanglement, temporal stabilization, and geometricnconsistency. Their combined effect enables stable and accurate zero-shot pose estimation in open-vocabulary scenarios.

5. Discussion

Our results demonstrate competitive performance on CAMERA25 and REAL275 under zero-shot conditions. The core innovation of ProM-Pose lies not in the novelty of individual modules, but in their strategic integration to solve the under-determined 9-DoF estimation problem in open-world settings. By anchoring the system with language-conditioned generative priors and refining it through temporal metric constraints, we provide a robust alternative to category-dependent models like Diff9D [36] and scale-limited frameworks like FoundationPose [35]. However, several challenges remain unresolved. First, the generative prior quality depends on the underlying diffusion model and may fail for highly irregular or articulated objects. Second, the framework struggles with texture-less regions where visual features become less discriminative. Third, cross-domain performance on REAL275 ($38.2\%$ at $5^{°},2\mathrm{cm}$) still lags behind synthetic results, indicating persistent sim-to-real gaps under strict thresholds. Key limitations include the following: RGB-D dependency restricting deployment to depth-enabled sensors, fixed temporal windows unable to handle prolonged occlusions, one-time prior generation vulnerable to poor initial frames, and modest inference speed (12 FPS) constraining real-time applications. Future work should prioritize monocular RGB extension, adaptive multi-hypothesis prior selection, and acceleration strategies.

6. Conclusions

This paper presents ProM-Pose, a unified cross-modal temporal perception framework for zero-shot 9-DoF object pose estimation from RGB-D observations. By integrating language-conditioned generative 3D shape priors as canonical geometric references, an asymmetric cross-modal attention mechanism for spatially aware fusion, and a decoupled pose decoding strategy with temporal refinement, ProM-Pose provides a novel system-level solution for 9-DoF modeling that transcends the boundaries of predefined categories and the limitations of isotropic scale assumptions found in current state-of-the-art foundations. Extensive experiments on CAMERA25 and REAL275 benchmarks demonstrate that our method achieves competitive or superior performance compared to state-of-the-art category-level approaches, particularly under strict pose thresholds. Qualitative evaluation on real-world logistics scenarios further validates temporal stability and robustness under occlusion and lighting variations. This work formalizes a structural perspective on zero-shot 9-DoF pose estimation, conceptualizing it as a multi-modal constraint satisfaction problem. Specifically, we show that semantic guidance, geometric consistency, and scale observability are intrinsically coupled rather than independently optimizable components. Language serves not merely as an auxiliary cue, but as a hypothesis selector that constrains the geometric solution space under open-vocabulary conditions. Geometric consistency provides the physical grounding necessary to stabilize metric reasoning, while temporal variation enhances scale observability by resolving the intrinsic coupling between rotation and anisotropic scale. From this viewpoint, reliable 9-DoF perception emerges from the coordinated interaction between semantic selection and metric constraints, rather than from isolated architectural modules. We believe this unified formulation offers a principled direction for open-world spatial perception, where semantic openness and metric reliability must coexist. Future work will focus on extending the framework to monocular RGB settings, incorporating adaptive prior refinement mechanisms, and accelerating inference for real-time robotic applications.