SGFNet: Semantic-Guided Fusion Network with Closed-Loop Feedback for RGB-Infrared Oriented Object Detection

Abstract

In oriented object detection from drone imagery, many existing RGB-infrared (RGB-IR) fusion methods derive modality weights from input statistics alone, without regard for downstream detection objectives. We present SGFNet, a Semantic-Guided Fusion Network that feeds detection-level semantics back into the fusion stage through learned importance masks. SGFNet comprises three modules: (1) a Frequency-aware Disentanglement Module (FDM) that separates high-frequency textures from low-frequency thermal structures through Laplacian and Gaussian filtering; (2) a Semantic-Guided Module (SGM) that generates P5-level semantic masks to steer fusion toward detection-critical regions; and (3) an Adaptive Geometric Convolution (AGC) whose rotation-aware sampling matches receptive fields to arbitrarily oriented objects. On the DroneVehicle benchmark (28,439 RGB-IR pairs, five vehicle categories), SGFNet achieves 82.0% mAP@0.5, surpassing the runner-up DMM by 3.2 percentage points while lowering mean angular error from 7.4° to 6.2° (−16%). Ablation analysis attributes the largest single-module gain (+1.7 pp) to the semantic feedback path.

1. Introduction

Many existing multimodal fusion networks determine fusion weights from input feature statistics rather than detection objectives [1,2,3], revealing a key limitation in current Red-Green-Blue–Infrared (RGB-IR) detection pipelines. In conventional pipelines, RGB-IR features are fused by attention mechanisms [4,5] or concatenation rules [6] and then fed to the detector. This feedforward design determines fusion weights without access to detection semantics, so the fusion network has no knowledge of which objects are most relevant, which regions require sharper boundaries, or which modality carries more reliable cues for a given scene.

This disconnect has practical consequences in UAV-based oriented object detection [7,8], where targets are represented with five degrees of freedom (center, width, height, angle) under diverse environmental conditions. In nighttime surveillance, IR should dominate for thermal-signature targets, yet static fusion mechanisms treat day and night scenes identically [4,5]. For densely packed vehicles, high-frequency RGB edges are important for boundary discrimination, yet fusion networks have no mechanism to request enhanced texture information. RGB-IR modalities offer complementary strengths: RGB provides rich textures under favorable lighting, while IR captures thermal patterns less sensitive to illumination changes [9,10], motivating RGB-IR fusion for reliable aerial vehicle detection. However, existing fusion methods lack access to detection-level feedback [4,11,12], creating a critical bottleneck: fusion weights are fixed with respect to downstream task requirements, preventing scene-adaptive allocation during inference. In safety-sensitive UAV applications—such as traffic monitoring, search-and-rescue, and infrastructure inspection—this task-agnostic fusion can cause missed detections of occluded vehicles or inaccurate orientation estimates for densely packed objects, compromising situational awareness.

Existing methods confirm that this limitation persists across all major fusion paradigms [4,6,11,12,13]. Current multimodal fusion methods are categorized into three paradigms, each with notable limitations. Early fusion concatenates raw inputs [6,9] but ignores semantic discrepancies between modalities. Late fusion combines modality-specific outputs [13] but lacks joint optimization. Intermediate fusion, the dominant paradigm [4,5,12], merges features through attention mechanisms but relies on static fusion mechanisms that determine weights based solely on input features—without semantic feedback from detection objectives, treating all regions with uniform strategies regardless of detection difficulty. Beyond fusion placement, angular precision is rarely evaluated. Most oriented detection methods report only mAP without quantifying angular estimation accuracy [8,14,15]. For precision-critical applications such as autonomous landing or vehicle trajectory prediction, angular errors of even a few degrees can cause mission failure. Yet no systematic analysis examines how fusion strategies affect angular predictions.

To address these limitations, we propose a closed-loop fusion paradigm in which detection semantics explicitly guide the fusion process. Detection-level features—particularly at the P5 level of the feature pyramid [16]—encode task-relevant information including object locations, semantic categories, and localization confidence. To this end, SGFNet integrates three modules: (1) a Frequency-aware Disentanglement Module (FDM) that separates high-frequency edges from low-frequency semantic structures via Laplacian and Gaussian filtering, achieving provable orthogonality ($\Phi \ge 0.85$); (2) a Semantic-Guided Module (SGM) that extracts semantic importance masks from P5 detection features to dynamically modulate fusion weights, establishing closed-loop feedback with validated weight amplification ratio $(1+\lambda )$; and (3) an Adaptive Geometric Convolution (AGC) that generates rotation-sensitive sampling grids to bridge multimodal fusion with angular precision.

The main contributions of this work are summarized as follows:

• We propose a closed-loop fusion paradigm where detection-level features explicitly modulate fusion weights, replacing the conventional feedforward information flow with bidirectional gradient pathways.
• We design a Frequency-aware Disentanglement Module (FDM) that decomposes cross-modal features into orthogonal high- and low-frequency components ($\Phi \ge 0.85$), reducing inter-modal interference before fusion.
• We introduce a Semantic-Guided Module (SGM) that extracts task-relevant semantic masks from P5 detection features, enabling dynamic fusion weight modulation with validated amplification ratio $(1+\lambda )$.
• We develop an Adaptive Geometric Convolution (AGC) that aligns receptive fields with predicted orientations, explicitly addressing angular precision in multimodal detection.
• Extensive experiments on DroneVehicle demonstrate state-of-the-art performance. Ablation studies validate the contribution of each module and the effectiveness of the closed-loop mechanism.

The remainder of this paper is organized as follows. Section 2 reviews related work on multimodal fusion and oriented detection. Section 3 presents the SGFNet architecture. Section 4 provides experimental validation. Section 5 discusses the results and limitations. Section 6 concludes with future directions.

2. Related Work

This section reviews multimodal fusion, oriented object detection, and semantic-guided approaches, highlighting the gaps that motivate our closed-loop fusion paradigm.

2.1. Multimodal Fusion for Object Detection

RGB-IR fusion methods are categorized into three paradigms based on fusion position. Early fusion concatenates raw images at network input [6,9], suffering from modality interference due to appearance discrepancies between RGB and IR. Late fusion combines modality-specific outputs through voting or averaging [13], but cannot exploit complementary information during feature learning.

Intermediate fusion has become the dominant paradigm, merging features through attention mechanisms at middle network layers. GAFF [4] employs cross-attention for adaptive weight computation, and CMAFF [5] integrates channel and spatial attention. Recent task-aware methods attempt to bridge fusion with detection objectives: TarDAL [11] employs adversarial learning, SeAFusion [1] incorporates auxiliary segmentation, and E2E-MFD [2] enables end-to-end gradient flow. State-space models provide efficient alternatives: DMM [12] applies Mamba architecture for linear-complexity cross-modal interaction. Beyond detection, frequency-domain decoupling has proven effective in related visual tasks: Dong et al. [17] propose learning temporal distribution and spatial correlation for universal moving object segmentation. Their divide-and-conquer strategy—separating temporal and spatial cues before integration—is conceptually parallel to FDM.

Despite architectural diversity, most methods remain feedforward: information flows from fusion to detection only. Attention mechanisms query “which features exhibit stronger activation?” rather than “which features help detection more?” This creates a semantic disconnect: fusion weights reflect input statistics rather than detection requirements. Table 1 summarizes this evolution and positions SGFNet’s bidirectional information flow. We next review oriented detection methods that address rotation representation and receptive-field alignment.

Table 1. Evolution of multimodal fusion methods. SGFNet is distinguished by its direct task awareness and bidirectional information flow through semantic feedback.

Category	Representative	Strategy	Task Aware	Flow
Early Fusion	Liu [6]	Concatenation	None	Feedforward
Late Fusion	Xu [13]	Output combine	None	Feedforward
Attention	GAFF [4], CMAFF [5]	Cross-attention	Input-adaptive	Feedforward
Task-Aware	TarDAL [11], E2E-MFD [2]	Joint optim.	Indirect	Feedforward
State-Space	DMM [12]	SSM interaction	None	Feedforward
SGFNet	Ours	Semantic-mod.	Direct	Bidirectional

2.2. Oriented Object Detection

Oriented object detection extends horizontal bounding boxes to five-degree-of-freedom representations $(x,y,w,h,\theta )$, essential for densely packed aerial scenes where axis-aligned boxes cause significant overlap. Existing methods are broadly categorized into anchor-based and anchor-free paradigms, each addressing distinct challenges in rotation representation.

Anchor-based methods generate rotated region proposals. RoI Transformer [7] applies spatial transformation to convert horizontal RoIs into oriented ones through learned affine parameters. Oriented R-CNN [8] proposes a midpoint offset representation for direct oriented proposal generation, achieving improved recall for elongated objects. S²A-Net [14] introduces feature alignment modules to address the misalignment between axis-aligned convolutions and rotated objects. These methods rely on dense anchor configurations, incurring computational overhead in high-density scenes.

Anchor-free alternatives eliminate anchor dependency through direct angle regression. DCL [15] addresses the angle periodicity problem by converting angular regression into a classification task with Gaussian label smoothing. GWD [18] and KLD [19] propose distribution-based loss functions that unify angle and box regression. D²Q-DETR [20] adopts a Transformer-based architecture with dual-query design separating classification and localization. H²RBox-v2 [21] learns horizontal-to-rotated conversion using only horizontal box annotations.

Both paradigms share a common limitation: standard convolutions impose axis-aligned receptive fields. For a 3 × 3 convolution applied to a 45° vehicle, approximately 50% of sampling points fall outside the object boundary. Deformable Convolutions [22] learn adaptive offsets to mitigate this issue, though offsets are optimized implicitly without explicit orientation awareness. ARC [23] explicitly rotates kernel banks based on predicted angles but requires discrete angle bins limiting continuous rotation support.

A gap persists: oriented detection methods operate predominantly on single RGB modalities. Multimodal fusion methods rarely evaluate angular precision and typically report only mAP, without quantifying orientation estimation accuracy. SGFNet addresses this gap through AGC. When $|\Delta \theta |<5°$, rotation-aware sampling achieves >95% object coverage.

2.3. Semantic-Guided Approaches

Semantic information provides task-relevant cues for guiding fusion weight allocation. Existing approaches can be categorized by their semantic source: external pre-trained models versus task-specific internal features.

External semantic guidance leverages independently trained models. SegMIF [3] employs a frozen segmentation network to generate category-aware masks for spatially varying fusion weights. CDDFuse [24] applies semantic-aware cross-attention derived from pre-trained encoders for base-detail decomposition. SDNet [25] incorporates auxiliary segmentation branches trained on separate datasets. PSFusion [26] progressively injects features from large-scale pre-trained vision models. SCAFNet [27] introduces semantic-guided cascade adaptive fusion for infrared small target detection, using rich semantic features to guide the spatial distribution of low-level features. More recently, PISENet [28] addresses salient object detection in optical remote sensing images through progressive interaction and saliency-guided enhancement, employing a hierarchical encoder that captures both global topological structures and local fine-grained details via cross-level feature fusion. These methods improve fusion quality but suffer from a fundamental limitation: guidance-task mismatch—the semantic features are optimized for their source tasks (segmentation, classification) rather than the target detection task, creating suboptimal alignment between guidance signals and detection requirements.

Task-aware fusion attempts to bridge this gap through joint optimization. TarDAL [11] employs adversarial learning to simultaneously optimize fusion quality and detection performance. E2E-MFD [2] proposes end-to-end training that back-propagates detection gradients through the fusion network. SeAFusion [1] introduces semantic-aware losses computed from detection outputs. These approaches move toward task awareness but retain an open-loop architecture: detection performance influences fusion training through loss gradients, yet detection-level features cannot directly modulate fusion weights during inference.

The distinction between open-loop and closed-loop lies in information flow direction: open-loop methods only propagate gradients from detection to fusion during training, while closed-loop methods additionally route detection features back to modulate fusion weights during both training and inference. This architectural difference determines whether task-relevant information can dynamically guide fusion decisions in deployment.

SGFNet introduces closed-loop semantic feedback through SGM. Unlike external guidance, SGM extracts semantic importance masks directly from P5 detection features—the same features explicitly optimized through detection loss. Unlike open-loop methods, SGM establishes a bidirectional pathway where detection semantics actively modulate fusion weights, enabling task-adaptive fusion that dynamically responds to scene-specific detection requirements.

3. Materials and Methods

This section presents the SGFNet framework for semantic-guided RGB-IR oriented object detection. The architecture comprises three core modules: (1) FDM, which separates RGB and IR features into orthogonal high-frequency and low-frequency components (Section 3.2), (2) SGM, which leverages P5-level detection semantics to modulate fusion weights through a closed-loop feedback mechanism (Section 3.3), and (3) AGC, which performs rotation-aware feature sampling for oriented bounding box prediction (Section 3.4). Finally, we describe the loss functions and two-stage training strategy (Section 3.5).

3.1. Problem Formulation and Network Overview

Problem Definition. Given a pair of aligned RGB-IR images ${\mathbf{I}}_{rgb}\in {\mathbb{R}}^{H\times W\times 3}$ and ${\mathbf{I}}_{ir}\in {\mathbb{R}}^{H\times W\times 1}$, the task is to predict a set of oriented bounding boxes $\mathcal{B}={\left\{(x_i,y_i,w_i,h_i,{\theta }_i,c_i)\right\}}_{i=1}^N$, where $(x_i,y_i)$ denotes the center coordinates, $(w_i,h_i)$ represents width and height, ${\theta }_i\in [-{\displaystyle \frac{\pi }{2}},{\displaystyle \frac{\pi }{2}})$ is the rotation angle, and $c_i\in \{1,\dots ,K\}$ is the object category among K classes. Network Architecture. As illustrated in Figure 1, SGFNet processes paired RGB-IR inputs through a dual-stream ResNet-50 [29] backbone extracting multi-scale features ${\mathbf{C}}_k$ ($k\in \{2,3,4,5\}$) from each modality, followed by an FPN [16] that produces pyramid features ${\mathbf{P}}_k\in {\mathbb{R}}^{256\times {\displaystyle \frac{H}{2^k}}\times {\displaystyle \frac{W}{2^k}}}$. The core processing applies frequency decomposition and semantic modulation:

$${\mathbf{F}}_k^{high},{\mathbf{F}}_k^{low}=\mathrm{FDM}\left({\mathbf{P}}_k\right),{\mathbf{W}}_k^{enh}=\mathrm{SGM}({\mathbf{F}}_k^{high},{\mathbf{F}}_k^{low},{\mathbf{M}}_{sem})$$

(1)

where ${\mathbf{M}}_{sem}$ is the semantic importance mask from P5 detection features. Finally, AGC produces oriented predictions $\mathcal{B}=\mathrm{AGC}\left({\mathbf{F}}_{fused}\right)=\left\{(x,y,w,h,\theta ,c)\right\}$. Closed-Loop Formulation. Unlike conventional feedforward fusion, SGFNet establishes a semantic feedback pathway through sequential computation and gradient-based optimization. In the forward pass, the semantic mask is derived from P5 features before fusion:

$$\begin{array}{cc}\hfill {\mathbf{P}}_5& ={\mathrm{FPN}}_5\left(\mathrm{Backbone}({\mathbf{I}}_{rgb},{\mathbf{I}}_{ir})\right)\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill {\mathbf{M}}_{sem}& =\sigma \left({\mathrm{Conv}}_{1\times 1}\left(\mathrm{BN}\left({\mathbf{P}}_5\right)\right)\right)\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\mathbf{W}}_k^{enh}& ={\mathbf{W}}_k\odot \left(1+\lambda ·{\mathrm{Up}}_k\left({\mathbf{M}}_{sem}\right)\right)\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill {\mathbf{F}}_{fused}& =\sum _m{\mathbf{W}}_m^{enh}\odot {\mathbf{F}}_m\hfill \end{array}$$

(5)

where Equations (2)–(5) execute sequentially; specifically, Equation (3) computes the semantic mask and Equation (4) performs semantic reweighting. within a single forward pass; no iterative solving is required. The “closed-loop” manifests during training: detection loss ${\mathcal{L}}_{det}$ back-propagates through the detection head and FPN to update ${\mathbf{P}}_5$, which in turn modifies ${\mathbf{M}}_{sem}$ and ${\mathbf{W}}_k^{enh}$. This gradient pathway ${\mathcal{L}}_{det}\to {\mathbf{P}}_5\to {\mathbf{M}}_{sem}\to {\mathbf{W}}^{enh}\to {\mathbf{F}}_{fused}$ enables the fusion module to learn task-adaptive weights optimized for detection (Figure 1, dashed arrows).

Figure 1. Overview of the SGFNet architecture. The network processes paired RGB-IR inputs through a dual-stream backbone, separates frequency components via FDM, modulates fusion weights using SGM’s semantic masks from P5 features (closed-loop feedback), and predicts oriented bounding boxes through AGC. Cyan solid arrows denote forward feature flow, while pink dashed arrows indicate the semantic feedback path from detection semantics to fusion modulation.

3.2. Frequency-Aware Disentanglement Module

RGB-IR modalities exhibit significant appearance differences in the spatial domain—RGB captures reflected light with rich texture and color while IR records emitted thermal radiation with characteristic intensity profiles. Direct spatial-domain fusion causes cross-modal interference: high-frequency RGB noise corrupts low-frequency IR structures [24,25]. Frequency-domain representations reveal structural correspondences—both modalities share low-frequency semantic layouts while differing primarily in high-frequency details—motivating frequency separation prior to fusion.

FDM applies dual-branch processing to extract orthogonal frequency components from the concatenated backbone features $\mathbf{F}\in {\mathbb{R}}^{C\times H\times W}$. The high-frequency branch isolates edges and fine textures through Laplacian filtering followed by learnable refinement:

$${\mathbf{F}}_{high}={\mathrm{Conv}}_{3\times 3}\left(\mathrm{ReLU}\left(\mathrm{BN}({\mathbf{K}}_L\ast \mathbf{F})\right)\right)$$

(6)

where the discrete Laplacian kernel is:

$${\mathbf{K}}_L=\left[\begin{array}{ccc}-1& -1& -1\\ -1& 8& -1\\ -1& -1& -1\end{array}\right]$$

(7)

The low-frequency branch extracts smooth semantic structures through Gaussian filtering followed by self-attention based global context modeling:

$${\mathbf{F}}_{low}={\mathbf{F}}_{smooth}+\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d_k}}}\right)\mathbf{V}$$

(8)

where the self-attention follows scaled dot-product attention [30], ${\mathbf{F}}_{smooth}={\mathcal{G}}_{\sigma }\ast \mathbf{F}$ is the Gaussian-smoothed feature map with $\sigma =1.5$, and $\mathbf{Q},\mathbf{K},\mathbf{V}\in {\mathbb{R}}^{N\times d_k}$ are query, key, value projections with $d_k=64$. The module comprises 2 layers and 4 attention heads. To ensure complete frequency separation, we remove the DC component: ${\mathbf{F}}_{high}\leftarrow {\mathbf{F}}_{high}-\mathrm{mean}\left({\mathbf{F}}_{high}\right)$. Figure 2 illustrates the frequency-domain perception decoupling pipeline.

Figure 2. Frequency-domain perception decoupling in FDM. High-frequency textures and low-frequency thermal structures are separated to reduce cross-modal interference before fusion. The green block denotes Gaussian filtering in the low-frequency branch, while the upper branch corresponds to Laplacian-based high-frequency extraction.

The frequency separation ensures minimal information redundancy between branches. We define the orthogonality metric:

$$\Phi ({\mathbf{F}}_1,{\mathbf{F}}_2)=1-{\displaystyle \frac{|⟨{\mathbf{F}}_1,{\mathbf{F}}_2⟩|}{\parallel {\mathbf{F}}_1\parallel ·\parallel {\mathbf{F}}_2\parallel }}$$

(9)

Proposition 1

(Frequency Orthogonality Bound). For FDM with 8-connected Laplacian high-pass kernel ${\mathbf{K}}_L$ and Gaussian low-pass kernel ${\mathcal{G}}_{\sigma }$ with $\sigma =1.5$, the orthogonality metric satisfies $\Phi ({\mathbf{F}}_{high},{\mathbf{F}}_{low})\ge 0.85$ for natural image features.

Proof. 

We analyze the spectral overlap between filter branches. In the discrete Fourier domain, the 8-connected Laplacian kernel has transfer function (Equation (10)):

$${\widehat{K}}_L(u,v)=8-2\text{cos}\left(2\pi u\right)-2\text{cos}\left(2\pi v\right)-2\text{cos}\left(2\pi (u+v)\right)-2\text{cos}\left(2\pi (u-v)\right)$$

(10)

which equals zero at DC $(u,v)=(0,0)$ and increases toward high frequencies. The Gaussian kernel has transfer function (Equation (11)):

$${\widehat{K}}_G(u,v)=exp\left(-2{\pi }^2{\sigma }^2(u^2+v^2)\right)$$

(11)

which equals unity at DC and decays exponentially.

We define the normalized spectral overlap as:

$$\eta ={\displaystyle \frac{{\displaystyle \sum _{u,v}{\left|{\widehat{K}}_L(u,v)\right|}^2·{\left|{\widehat{K}}_G(u,v)\right|}^2}}{{\displaystyle \sqrt{\sum _{u,v}{\left|{\widehat{K}}_L(u,v)\right|}^4}·\sqrt{\sum _{u,v}{\left|{\widehat{K}}_G(u,v)\right|}^4}}}}$$

(12)

Evaluating Equation (12) over the discrete frequency grid $u,v\in [-0.5,0.5)$ at resolution $64\times 64$ with $\sigma =1.5$ yields $\eta =0.127$, corresponding to spectral overlap concentrated in a narrow annular band at intermediate frequencies $\left|\omega \right|\in [0.08,0.15]$ cycles/pixel (7% of the total spectral area).

By Parseval’s theorem, the inner product in the spatial domain equals the inner product in the frequency domain: $⟨{\mathbf{F}}_{high},{\mathbf{F}}_{low}⟩=⟨{\widehat{K}}_L·\widehat{F},{\widehat{K}}_G·\widehat{F}⟩$. Combined with DC removal (${\mathbf{F}}_{high}\leftarrow {\mathbf{F}}_{high}-\mathrm{mean}\left({\mathbf{F}}_{high}\right)$), the effective overlap reduces to ${\eta }_{\mathrm{eff}}\le 0.127$. Therefore, $\Phi =1-{\displaystyle \frac{|⟨{\mathbf{F}}_{high},{\mathbf{F}}_{low}⟩|}{\parallel {\mathbf{F}}_{high}\parallel ·\parallel {\mathbf{F}}_{low}\parallel }}\ge 1-{\eta }_{\mathrm{eff}}\ge 0.85$.    □

We verify this bound over a range of $\sigma$ values: computing Equation (12) for $\sigma \in \{1.0,1.5,2.0,2.5\}$ yields $\eta =\{0.183,0.127,0.094,0.071\}$, confirming that larger $\sigma$ strengthens separation. At $\sigma =1.0$, $\Phi \ge 0.82$ falls below the 0.85 threshold, justifying our choice of $\sigma =1.5$.

Empirical validation on DroneVehicle yields $\Phi =0.873\pm 0.021$, exceeding the theoretical bound. A dedicated sensitivity analysis in Section 4 examines the effect of the orthogonality threshold on detection performance: the threshold 0.85 achieves the best mAP (82.0%), as looser constraints (0.70, 0.80) permit residual cross-modal interference while a stricter constraint (0.90) over-regularizes and suppresses discriminative features, confirming 0.85 as the optimal balance between decorrelation and information preservation. Proposition 1 provides the theoretical bound used in our regularization.

To maintain frequency separation during training, we introduce an orthogonality regularization loss:

$${\mathcal{L}}_{orth}=max(0,0.85-\Phi ({\mathbf{F}}_{high},{\mathbf{F}}_{low}))$$

(13)

3.3. Semantic-Guided Module

Choice of semantic source level. We select P5 as the semantic guidance source for two reasons. First, P5 features undergo the most extensive spatial downsampling ($32\times$), yielding a global receptive field that spans the entire image. This enables the semantic mask to capture where targets are located rather than local texture details—the task-relevant information needed for fusion modulation. Second, P5 is the topmost FPN level and receives the strongest supervision from detection loss, making its features the most task-aligned among all pyramid levels. The semantic-level ablation in Section 4 validates this choice empirically.

The SGM establishes closed-loop semantic feedback from detection to fusion. Existing methods determine fusion weights through cross-modal attention:

$$\mathbf{W}=\sigma \left({\mathbf{W}}_q{\mathbf{F}}_{rgb}^{\top }{\mathbf{W}}_k{\mathbf{F}}_{ir}/\sqrt{d}\right)$$

(14)

where ${\mathbf{W}}_q,{\mathbf{W}}_k\in {\mathbb{R}}^{d\times C}$ are learnable projections. This input-adaptive approach queries “which modality features exhibit stronger activation patterns?” rather than “which modality features contribute most to detection?” P5 features in Feature Pyramid Networks encode exactly this task-relevant information, as they are explicitly optimized through detection loss.

SGM extracts semantic importance masks from P5 detection features. The P5 feature map ${\mathbf{P}}_5\in {\mathbb{R}}^{C_5\times {\displaystyle \frac{H}{32}}\times {\displaystyle \frac{W}{32}}}$ encodes high-level detection semantics. The semantic mask is generated through:

$${\mathbf{M}}_{sem}=\sigma \left({\mathbf{W}}_m·\mathrm{BN}\left({\mathbf{P}}_5\right)+{\mathbf{b}}_m\right)\in {[0,1]}^{{\displaystyle \frac{H}{32}}\times {\displaystyle \frac{W}{32}}}$$

(15)

where ${\mathbf{W}}_m\in {\mathbb{R}}^{1\times C_5}$ and ${\mathbf{b}}_m$ are learnable parameters and $\sigma (·)$ is the sigmoid activation. High mask values indicate object regions requiring enhanced fusion attention, while low values indicate background. The mask is learned end-to-end through detection loss without auxiliary annotations.

To modulate fusion weights across FPN levels, the semantic mask is bilinearly upsampled to match each level’s resolution. Fusion weights are modulated through:

$${\mathbf{W}}_k^{enh}={\mathbf{W}}_k\odot (1+\lambda ·{\mathbf{M}}_{sem}^k)$$

(16)

where $\lambda =0.1$ is the modulation coefficient and ⊙ represents element-wise multiplication. Figure 3 shows the semantic-guided module architecture.

Figure 3. Semantic-Guided Module (SGM). P5 detection features generate semantic masks that modulate fusion weights across FPN levels for task-driven fusion.

Theorem 1

(Semantic Weight Amplification). For target regions where ${\mathbf{M}}_{sem}\approx 1$ and background regions where ${\mathbf{M}}_{sem}\approx 0$, the enhanced fusion weight ratio satisfies:

$${\displaystyle \frac{{\mathbf{W}}_{enh}^{target}}{{\mathbf{W}}_{enh}^{bg}}}=1+\lambda$$

(17)

Proof. 

For target regions where ${\mathbf{M}}_{sem}\approx 1$, we have ${\mathbf{W}}_{enh}^{target}=\mathbf{W}\odot (1+\lambda ·1)=\mathbf{W}·(1+\lambda )$. For background regions where ${\mathbf{M}}_{sem}\approx 0$, we have ${\mathbf{W}}_{enh}^{bg}=\mathbf{W}\odot (1+\lambda ·0)=\mathbf{W}$. Taking the ratio yields ${\mathbf{W}}_{enh}^{target}/{\mathbf{W}}_{enh}^{bg}=(1+\lambda )$, which equals 1.10 when $\lambda =0.1$.    □

With $\lambda =0.1$, detected object regions receive 10% higher fusion weights. Empirical validation yields a measured weight ratio of $1.098\pm 0.012$, within 2% of the theoretical prediction.

The closed-loop design enables bidirectional optimization. The total gradient includes both a direct path and a semantic feedback path:

$${\displaystyle \frac{\mathsf{\partial }{\mathcal{L}}_{det}}{\mathsf{\partial }{\theta }_F}}=\underset{\mathrm{Direct}\mathrm{path}}{\underbrace{{\displaystyle \frac{\mathsf{\partial }{\mathcal{L}}_{det}}{\mathsf{\partial }{\mathbf{F}}_{fused}}}·{\displaystyle \frac{\mathsf{\partial }{\mathbf{F}}_{fused}}{\mathsf{\partial }{\theta }_F}}}}+\underset{\mathrm{Semantic}\mathrm{feedback}\mathrm{path}}{\underbrace{{\displaystyle \frac{\mathsf{\partial }{\mathcal{L}}_{det}}{\mathsf{\partial }{\mathbf{M}}_{sem}}}·{\displaystyle \frac{\mathsf{\partial }{\mathbf{M}}_{sem}}{\mathsf{\partial }{\mathbf{P}}_5}}·{\displaystyle \frac{\mathsf{\partial }{\mathbf{P}}_5}{\mathsf{\partial }{\mathbf{F}}_{fused}}}·{\displaystyle \frac{\mathsf{\partial }{\mathbf{F}}_{fused}}{\mathsf{\partial }{\theta }_F}}}}$$

(18)

This gradient chain establishes a feedback loop where detection performance directly shapes fusion behavior.

3.4. Adaptive Geometric Convolution

Oriented objects span the full 360-degree rotation range, yet standard convolutions impose fixed axis-aligned receptive fields. For a 3 × 3 convolution applied to a 45° rotated vehicle, approximately 50% of sampling points fall on background pixels, contaminating angular features. AGC addresses this by rotating the sampling grid according to predicted orientation angles.

Sine–cosine parameterization avoids discontinuity at boundary values: ${\theta }_{pred}=\mathrm{atan}2({\text{sin}}_{pred},{\text{cos}}_{pred})$, where separate heads predict ${\text{sin}}_{pred}$ and ${\text{cos}}_{pred}$. Given the predicted angle $\theta$, we construct a 2D rotation matrix and transform the standard 3 × 3 sampling grid:

$$\mathbf{R}\left(\theta \right)=\left[\begin{array}{cc}\text{cos}\theta & -\text{sin}\theta \\ \text{sin}\theta & \text{cos}\theta \end{array}\right],{\mathbf{G}}^{\prime }=\{\mathbf{R}\left(\theta \right)·\mathbf{p}|\mathbf{p}\in \mathbf{G}\}$$

(19)

The rotated grid positions ${\mathbf{G}}^{\prime }$ generally fall at sub-pixel locations; bilinear interpolation samples feature values. Boundary handling and differentiability. When the rotated sampling grid ${\mathbf{G}}^{\prime }$ extends beyond the feature map boundary, out-of-bound positions are zero-padded: $F\left(p\right)=0$ for $p\notin [0,H)\times [0,W)$, implemented via PyTorch 2.1.0’s grid_sample with padding_mode=’zeros’. The bilinear interpolation is fully differentiable: for a sampling point $p=(p_y,p_x)$ with four integer neighbors $\left\{q_{ij}\right\}$, gradients decompose as $\mathsf{\partial }{F}_{\mathrm{agc}}/\mathsf{\partial }F\left(q_{ij}\right)=w_{ij}$ (bilinear coefficients) and $\mathsf{\partial }{F}_{\mathrm{agc}}/\mathsf{\partial }p={\sum }_{ij}(\mathsf{\partial }{w}_{ij}/\mathsf{\partial }p)·F\left(q_{ij}\right)$, enabling end-to-end training of the angle predictor through standard back-propagation.

For a vehicle rotated by ${\theta }_{gt}$, the proportion of sampling points within the object region is $P_{object}\approx 1-{\displaystyle \frac{2|\Delta \theta |}{\pi }}·\rho$, where $\rho$ is aspect ratio. For $|\Delta \theta |<5°$ and typical aspect ratios ($\rho \approx 3$), $P_{object}>0.95$. Compared to DCN (18 offset parameters) and ARC (discrete kernel banks), AGC requires only 1 parameter per location while supporting continuous rotation—enabling deployment on resource-constrained platforms. Table 2 provides a concise method-level comparison, and Figure 4 visualizes the adaptive geometric convolution for rotation-aware sampling.

Table 2. Comparison of geometric convolution methods.

Method	Params/loc.	FLOPs	Angle-Aware	Continuous
Standard Conv	0	1.0×	No	N/A
DCN [22]	18	1.2×	Implicit	Yes
ARC [23]	K (discrete)	1.5–2.0×	Yes	No
AGC (Ours)	1	1.3×	Yes	Yes

Figure 4. Adaptive Geometric Convolution (AGC). The sampling grid is rotated according to the predicted angle to align receptive fields with oriented objects. Solid arrows indicate feature-flow direction, and dashed arrows indicate angle-conditioned rotated sampling correspondence.

3.5. Loss Functions and Training Strategy

We define the total loss by combining detection accuracy with auxiliary constraints:

$${\mathcal{L}}_{total}={\mathcal{L}}_{det}+{\lambda }_{orth}·{\mathcal{L}}_{orth}+{\lambda }_{angle}·{\mathcal{L}}_{angle}$$

(20)

where ${\mathcal{L}}_{det}={\mathcal{L}}_{cls}+{\mathcal{L}}_{reg}$ with focal loss [31] for classification and Smooth L1 [32] for box regression. The angle loss uses sine-cosine representation:

$${\mathcal{L}}_{angle}=\mathrm{SmoothL}1({\text{sin}}_{pred},{\text{sin}}_{gt})+\mathrm{SmoothL}1({\text{cos}}_{pred},{\text{cos}}_{gt})$$

(21)

We set ${\lambda }_{orth}=0.1$ and ${\lambda }_{angle}=1.5$.

Training follows a two-stage strategy: Stage 1 (Epochs 1–10) freezes the semantic mask generator to prevent the “cold start” problem where random P5 features would generate meaningless masks. Stage 2 (Epochs 11–60) jointly optimizes all parameters with full loss, activating the semantic feedback loop.

Theorem 2

(Convergence Guarantee). Under Lipschitz continuity of ${\mathcal{L}}_{total}$ with constant L, learning rate $\eta <1/\left(2L\right)$, and two-stage training, the closed-loop optimization converges to a local minimum without oscillation.

Proof Sketch. 

In Stage 1, with SGM frozen, the optimization is equivalent to standard feed-forward training, which converges under standard assumptions. In Stage 2, the closed-loop gradient decomposes as ${\nabla }_{\theta }\mathcal{L}={\nabla }^{direct}+{\nabla }^{feedback}$. Since $\parallel {\nabla }^{feedback}\parallel \le \lambda \parallel {\nabla }^{direct}\parallel$ (bounded by modulation strength), the total gradient remains Lipschitz continuous. With $\eta <1/\left(2L\right)$, the descent lemma guarantees $\mathcal{L}\left({\theta }^{t+1}\right)\le \mathcal{L}\left({\theta }^t\right)-{\displaystyle \frac{\eta }{2}}{\parallel \nabla \mathcal{L}\parallel }^2$, ensuring monotonic convergence. □

4. Experiments

This section presents the experimental setup, comparisons with state-of-the-art methods, and ablation studies.

4.1. Experimental Setup

Dataset. To evaluate the effectiveness of SGFNet, we conduct experiments on the DroneVehicle dataset [10], the largest publicly available benchmark for multimodal oriented vehicle detection in drone imagery. The dataset contains 28,439 aligned RGB-IR image pairs captured from drone platforms at 840 × 712 resolution. The dataset covers five vehicle categories (car, truck, bus, van, and freight car) with oriented bounding box annotations, presenting significant challenges in scale variation, dense packing (up to 80 vehicles per image), and illumination diversity (daytime, nighttime, and twilight conditions). The multimodal nature of DroneVehicle is essential for evaluating fusion methods: nighttime scenes demonstrate IR dominance while daytime dense scenes show RGB’s advantage in fine boundary discrimination.

Evaluation Metrics. We adopt mean Average Precision (mAP) as the primary metric at IoU thresholds of 0.5, 0.75, and 0.5:0.95. For angular precision, we additionally report: (1) mean Absolute Angular Error (ΔAngle); (2) Angular Accuracy at 5° (AA@5°) and 10° (AA@10°). All experiments are repeated 3 times with different random seeds; we report mean ± standard deviation and conduct paired t-tests for significance analysis ($p<0.05$).

Implementation Details. SGFNet is implemented in PyTorch 2.1.0 and trained on 2× NVIDIA RTX 3090 GPUs. ResNet-50 [29] pretrained on ImageNet serves as the shared backbone. FPN [16] generates features at levels P2–P5. FDM uses an 8-connected Laplacian kernel, Gaussian $\sigma =1.5$, and a lightweight Transformer encoder (2 layers, 4 attention heads, 256 dimensions). SGM uses a single $1\times 1$ convolution with $\lambda =0.1$. AGC employs sine-cosine angle parameterization. Training follows a two-stage protocol: Stage 1 (epochs 1–10) with learning rate $1\times {10}^{-4}$; Stage 2 (epochs 11–60) with cosine annealing. AdamW optimizer with weight decay 0.05 and batch size 16. Input images are resized to $640\times 640$ using direct resizing to avoid padding artifacts; the aspect ratio change from 1.18:1 to 1:1 introduces minor spatial distortion, but all compared methods use identical preprocessing, ensuring fair comparison. Images are then augmented with random horizontal flips. To support reproducibility, the source code, pre-trained weights, and evaluation scripts are available from the corresponding author upon reasonable request.

Hyperparameter Selection. Key hyperparameters are determined via grid search on the validation set. SGM modulation strength $\lambda$ is selected from $\{0.00,0.05,0.10,0.15,0.20\}$, with $\lambda =0.10$ as the optimum. Gaussian sigma $\sigma$ is chosen from $\{1.0,1.5,2.0\}$: $\sigma =1.0$ yields insufficient separation ($\Phi =0.82$), while $\sigma =2.0$ over-smooths (−0.5% mAP); $\sigma =1.5$ provides the best balance. The orthogonality threshold is selected from $\{0.70,0.80,0.85,0.90\}$, with 0.85 optimal. Angle loss weight ${\lambda }_{angle}$ is chosen from $\{0.5,1.0,1.5,2.0,2.5\}$, with ${\lambda }_{angle}=1.5$ as optimal. All hyperparameters are selected on the validation split and fixed for test evaluation.

4.2. Comparison with Leading Methods

We compare SGFNet against single-modality baselines and state-of-the-art multimodal fusion methods. All methods use identical data splits and augmentation strategies.

Table 3 shows that SGFNet achieves 82.0% mAP@0.5, exceeding DMM by 3.2 percentage points. The improvement is statistically significant ($p<0.001$) with identical data splits and augmentation settings. Gains remain at stricter IoU thresholds (mAP@0.75: +2.6 pp, mAP@0.5:0.95: +2.3 pp), indicating improved localization precision in addition to recall.

Table 3. Comparison on DroneVehicle oriented object detection. ↑: higher is better; ↓: lower is better. Best in bold; second-best underlined. Results for comparison methods are reproduced under identical settings.

Method	Modal.	mAP@0.5 ↑	mAP@0.75 ↑	mAP@0.5:0.95 ↑	$\mathbf{\Delta }$Angle ↓	AA@5° ↑	AA@10° ↑
Single-Modality Baselines
Oriented R-CNN [8]	RGB	62.3 ± 0.4	41.5 ± 0.3	38.2 ± 0.3	15.3°	42.1%	68.3%
Oriented R-CNN [8]	IR	58.7 ± 0.5	38.2 ± 0.4	35.1 ± 0.3	16.8°	38.7%	62.5%
RGB-IR Fusion Methods
Halfway Fusion [6]	RGB-IR	71.2 ± 0.4	48.3 ± 0.4	44.5 ± 0.3	11.2°	58.3%	76.8%
AR-CNN [33]	RGB-IR	73.5 ± 0.4	50.1 ± 0.3	46.2 ± 0.3	10.1°	65.2%	81.4%
MBNet [34]	RGB-IR	75.3 ± 0.3	51.8 ± 0.4	47.8 ± 0.3	9.3°	71.8%	84.6%
TSFADet [35]	RGB-IR	76.8 ± 0.3	53.2 ± 0.3	49.1 ± 0.3	8.5°	76.4%	87.9%
C2Former [36]	RGB-IR	78.5 ± 0.3	55.6 ± 0.3	51.2 ± 0.3	7.6°	82.3%	90.8%
DMM [12]	RGB-IR	78.8 ± 0.3	55.9 ± 0.3	51.5 ± 0.3	7.4°	83.7%	91.5%
SGFNet (Ours)	RGB-IR	82.0 ± 0.3	58.5 ± 0.3	53.8 ± 0.2	6.2°	92.4%	96.8%

For angular precision, Table 3 reports that ΔAngle drops from 7.4° (DMM) to 6.2° (SGFNet), a 16% relative reduction. AA@5° increases from 83.7% to 92.4% (+8.7 pp) and AA@10° from 91.5% to 96.8% (+5.3 pp). Relative to RGB-only Oriented R-CNN (15.3°), SGFNet reduces mean angular error by 59%. Figure 5 visualizes the overall comparison across methods.

Figure 5. Visual comparison of detection results on DroneVehicle. Red circles highlight vehicles detected by SGFNet but missed by DMM. Yellow arrows indicate improved orientation accuracy. Green boxes denote oriented vehicle detections, and blue boxes denote the zoom-in regions shown in the bottom row. Zoom-in patches (bottom row) show detailed boundary and angle alignment differences. SGFNet achieves the best overall accuracy and angular precision among representative RGB-IR fusion methods.

Scene-Conditioned Analysis. To evaluate robustness across illumination conditions, we partition the DroneVehicle test set into daytime (52%), nighttime (35%), and twilight (13%) subsets and report per-condition performance in Table 4.

Table 4. Scene-conditioned performance on DroneVehicle test subsets. Overall mAP is computed globally on the full test set rather than as a weighted average of subset values, because AP is a non-linear function of the precision-recall curve. Best in bold. ↑: higher is better.

Condition	Subset %	DMM mAP	SGFNet mAP ↑	$\mathbf{\Delta }$	$\mathbf{\Delta }$Angle
Daytime	52%	80.5	84.2	+3.7	5.8°
Nighttime	35%	77.2	80.1	+2.9	6.5°
Twilight	13%	72.8	74.8	+2.0	8.1°
Overall	100%	78.8	82.0	+3.2	6.2°

SGFNet consistently outperforms DMM across all conditions. Daytime scenes yield the largest gain (+3.7 pp), where rich RGB textures provide strong high-frequency features for FDM. Nighttime gains (+2.9 pp) demonstrate that SGM leverages IR-dominant semantics. Twilight scenes show the smallest improvement (+2.0 pp), consistent with the failure analysis in Section 5: thermal crossover degrades both modality streams.

4.2.1. Per-Category Analysis

Table 5 shows consistent gains across all categories. Freight Car yields the largest improvement (+4.1 over DMM), suggesting that semantic guidance benefits long-tailed classes. Elongated objects (Truck +3.4, Bus +3.6) show larger gains, consistent with AGC’s rotation-aware sampling.

Table 5. Per-category AP@0.5 comparison on DroneVehicle. mAP@0.5 in Table 3 is instance-weighted; the unweighted average here differs because Car instances account for ∼62% of the dataset. Best in bold.

Category	RGB-Only	IR-Only	TSFADet	C2Former	DMM	SGFNet (Ours)
Car	68.2	62.5	79.5	81.0	81.4	84.5
Truck	56.3	54.8	73.8	75.2	75.8	79.2
Bus	52.7	58.3	74.2	76.0	76.5	80.1
Van	54.1	52.3	72.1	74.0	74.6	78.3
Freight Car	45.2	48.6	66.8	69.2	70.1	74.2
Unweighted Avg	55.3	55.3	73.3	75.1	75.7	79.3

4.2.2. Per-Category Angular Error

Table 6 confirms that angular errors decrease with higher aspect ratios, supporting AGC’s effectiveness on elongated objects.

Table 6. Per-category angular error (ΔAngle in degrees) on DroneVehicle.

Category	Aspect Ratio	DMM	SGFNet (Ours)	Improvement
Car	1.3–1.8	8.0°	6.8°	−15%
Truck	2.5–4.0	6.5°	5.2°	−20%
Bus	3.5–5.0	5.8°	4.5°	−22%
Van	1.8–2.5	7.1°	5.8°	−18%
Freight Car	4.0–6.0	5.2°	4.0°	−23%
Weighted Avg	–	7.4°	6.2°	−16%

4.3. Ablation Studies

Component Ablation. Table 7 presents the component ablation results. All improvements over baseline are statistically significant ($p<0.001$), based on 3 runs per configuration.

Table 7. Ablation study with all module combinations. $\Phi$: frequency orthogonality metric. Best in bold. ↑: higher is better; ↓: lower is better.

Configuration	mAP@0.5 ↑	$\mathbf{\Delta }$Angle ↓	$\mathbf{\Phi }$ ↑	$\mathbf{\Delta }$mAP
Baseline (Concat)	78.2 ± 0.3	9.8°	0.42	–
Single Module
+ FDM	79.5 ± 0.3	8.4°	0.87	+1.3
+ SGM	79.1 ± 0.3	8.9°	0.42	+0.9
+ AGC	78.9 ± 0.3	8.1°	0.42	+0.7
Two Modules
+ FDM + SGM	81.2 ± 0.2	7.1°	0.87	+3.0
+ FDM + AGC	80.1 ± 0.3	6.8°	0.87	+1.9
+ SGM + AGC	80.4 ± 0.3	7.5°	0.42	+2.2
Full Model
SGFNet (Full, Ours)	82.0 ± 0.3	6.2°	0.87	+3.8

Table 7 indicates both individual contributions and interaction effects. FDM improves mAP by +1.3 pp with frequency separation ($\Phi$: 0.42 → 0.87). SGM alone adds +0.9 pp, whereas with FDM it adds +1.7 pp (FDM + SGM vs. FDM), suggesting that semantic modulation benefits from frequency separation. SGM contributes 45% of the total gain, confirming the central role of semantic feedback. AGC adds +0.8 pp mAP while reducing ΔAngle from 7.1° to 6.2° (13% reduction). Figure 6 visualizes the ablation trends across module combinations.

Figure 6. Ablation experiment results for FDM, SGM, and AGC. The combined model yields the strongest gains in mAP and angular precision. Green boxes denote car detections, and purple boxes denote van detections for category-wise visual distinction.

Hyperparameter Sensitivity. We report the effects of SGM modulation strength $\lambda$, orthogonality threshold, and angle loss weight ${\lambda }_{angle}$ in Table 8, Table 9, and Table 10, respectively.

Table 8. Effect of SGM modulation strength $\lambda$ on detection performance. Best in bold. ↑: higher is better; ↓: lower is better.

$\mathit{\lambda }$	mAP@0.5 ↑	$\mathbf{\Delta }$Angle ↓	Weight Ratio	Notes
0.00	79.5	8.4°	1.00	No modulation
0.05	80.8	7.4°	1.05	Mild enhancement
0.10	82.0	6.2°	1.10	Optimal
0.15	81.6	6.4°	1.15	Slight over-modulation
0.20	80.9	6.8°	1.20	Background under-weighted

Table 9. Effect of orthogonality threshold on performance. Best in bold. ↑: higher is better.

Threshold	mAP@0.5	Achieved $\mathbf{\Phi }$	Notes
0.70	81.3	0.78	Loose constraint
0.80	81.7	0.84	Moderate constraint
0.85	82.0	0.87	Optimal (matches theory)
0.90	81.5	0.89	Over-constrained

Table 10. Effect of angle loss weight ${\lambda }_{angle}$ on performance. Best in bold. ↑: higher is better.

${\mathit{\lambda }}_{\mathit{angle}}$	mAP@0.5	$\mathbf{\Delta }$Angle	AA@5°
0.5	82.1	7.8°	84.3%
1.0	82.0	6.9°	89.1%
1.5	82.0	6.2°	92.4%
2.0	81.7	6.0°	93.1%
2.5	81.2	5.8°	93.8%

Table 8 indicates that $\lambda =0.10$ achieves the best balance. The measured weight ratio ($1.098\pm 0.012$) closely matches the theoretical prediction ($1+\lambda =1.10$), validating Theorem 1. Table 9 shows that a threshold of 0.85 yields the best mAP with an achieved $\Phi$ of 0.87. Table 10 indicates that ${\lambda }_{angle}=1.5$ provides the best balance between detection accuracy and angular precision.

Two-Stage Training Ablation. Table 11 validates the necessity of two-stage training by comparing different strategies. End-to-end training from Epoch 1 suffers from the cold start problem: random P5 features produce noisy semantic masks that destabilize early optimization, reflected in higher variance (±0.5 vs. ±0.3) and 1.7 pp lower mAP. Freezing SGM for 10 epochs provides sufficient warm-up for reliable P5 representations.

Table 11. Two-stage training strategy ablation. S1: SGM frozen; S2: joint optimization. Best in bold. ↑: higher is better; ↓: lower is better.

Strategy	mAP@0.5 ↑	$\mathbf{\Delta }$Angle ↓	Convergence (Ep)	Notes
End-to-end (no freeze)	80.3 ± 0.5	7.5°	48	Cold start
S1(5ep) + S2(55ep)	81.5 ± 0.3	6.6°	42	Partial warm-up
S1(10ep) + S2(50ep)	82.0 ± 0.3	6.2°	38	Optimal
S1(15ep) + S2(45ep)	81.8 ± 0.3	6.3°	40	Over-frozen
S1(20ep) + S2(40ep)	81.5 ± 0.3	6.4°	43	Reduced joint training

Semantic Guidance Source Comparison. Table 12 compares different semantic guidance strategies under the same framework (FDM + AGC fixed). External guidance from a pretrained segmentation model (DeepLabV3) adds 12.3M parameters but only +0.7 pp mAP because the segmentation features are not jointly optimized for detection; its ΔAngle (7.0°) is slightly worse than the no-guidance baseline (6.8°) because pretrained segmentation features emphasize semantic boundaries rather than object orientations, introducing conflicting gradients to the angle predictor. An auxiliary segmentation head offers partial joint optimization but requires pixel-level segmentation annotations. Our internal P5 guidance achieves the highest mAP with the fewest extra parameters and no auxiliary labels.

Table 12. Comparison of semantic guidance sources. All models use FDM + AGC; only the semantic guidance varies. Best in bold. ↑: higher is better; ↓: lower is better.

Guidance Source	mAP@0.5 ↑	$\mathbf{\Delta }$Angle ↓	Extra Params	Extra Labels
None (FDM + AGC only)	80.1 ± 0.3	6.8°	—	—
External (pretrained DeepLabV3)	80.8 ± 0.4	7.0°	+12.3 M	Seg masks
External (auxiliary seg head)	81.2 ± 0.3	6.7°	+5.8 M	Seg masks
Internal P5 (Ours)	82.0 ± 0.3	6.2°	+2.1 M	None

Semantic Level Selection. Table 13 validates the choice of P5 as semantic guidance source. Lower pyramid levels (P3, P4) have smaller receptive fields and weaker detection supervision, resulting in less effective semantic masks. Combining P4 and P5 yields marginal improvement over P5 alone while doubling the mask computation cost.

Table 13. Effect of FPN level for semantic guidance extraction. Best in bold. ↑: higher is better; ↓: lower is better.

Level	Receptive Field	mAP@0.5 ↑	$\mathbf{\Delta }$Angle ↓	Notes
P3 ($8\times$ downsample)	Local	80.1 ± 0.3	7.5°	Texture-dominant
P4 ($16\times$ downsample)	Moderate	81.0 ± 0.3	7.0°	Mixed semantics
P5 ($\mathbf{32}\times$ downsample)	Global	82.0 ± 0.3	6.2°	Task-aligned
P4 + P5 (concatenated)	Global	81.8 ± 0.3	6.4°	Marginal gain, +cost

FDM Filter Method Comparison. Table 14 compares different frequency separation approaches in FDM. Laplacian–Gaussian achieves the highest orthogonality ($\Phi =0.87$), which enables stronger semantic modulation downstream. We focus on single-pass spatial filters for computational efficiency; multi-resolution methods (e.g., wavelet decomposition) offer richer frequency representations but incur higher cost due to multi-level decomposition and reconstruction, and are deferred to future work.

Table 14. Comparison of frequency separation filters in FDM. Best in bold. ↑: higher is better.

Filter Method	mAP@0.5 ↑	$\mathbf{\Phi }$ ↑	FLOPs (G)	Notes
Sobel HF + Mean LF	80.8 ± 0.3	0.79	127.8	Directional bias
DoG bandpass	81.2 ± 0.3	0.81	128.1	Fixed bandwidth
Learned 3 × 3	81.4 ± 0.3	0.83	129.2	Data-dependent
Laplacian + Gaussian (Ours)	82.0 ± 0.3	0.87	128.5	Isotropic, stable

Self-Attention Configuration. Table 15 reports the effect of different self-attention configurations in the low-frequency branch of FDM.

Table 15. Effect of self-attention configuration in FDM low-frequency branch. Best in bold. ↑: higher is better.

Layers	Heads	Dim	mAP@0.5 ↑	FLOPs (G)	FPS ↑
1	2	128	81.2 ± 0.3	124.8	31.5
1	4	256	81.5 ± 0.3	126.2	30.2
2	4	256	82.0 ± 0.3	128.5	28.3
4	4	256	82.1 ± 0.3	135.3	23.7
2	8	256	81.8 ± 0.3	130.1	27.1

Two layers with four heads offer the best accuracy–efficiency trade-off: 4 layers yield only +0.1 pp mAP while reducing FPS by 16%.

4.4. Computational Efficiency and Failure Analysis

4.4.1. Efficiency Comparison

Table 16 shows that SGFNet offers a favorable accuracy–efficiency trade-off: 25% fewer FLOPs than DMM (128.5 vs. 172.8 G), 59% faster inference (28.3 vs. 17.8 FPS), and 39% higher mAP/GFLOP ratio (0.64 vs. 0.46).

Table 16. Computational efficiency comparison. FLOPs include the full pipeline (backbone + FPN + FDM + SGM + AGC + detection head). FPS measured on a single NVIDIA RTX 3090 with FP32 precision, batch size 1, $640\times 640$ input, excluding data preprocessing. Reported FPS is the median of 1000 runs after 100 warm-up iterations. Best in bold. ↑: higher is better; ↓: lower is better.

Method	Params (M) ↓	FLOPs (G) ↓	FPS ↑	mAP@0.5 ↑	mAP/GFLOP ↑
Halfway Fusion	38.4	112.6	32.1	71.2	0.63
AR-CNN	42.1	128.3	26.4	73.5	0.57
MBNet	46.5	138.7	24.2	75.3	0.54
TSFADet	49.2	152.4	21.3	76.8	0.50
C2Former	52.8	168.2	18.5	78.5	0.47
DMM	54.1	172.8	17.8	78.8	0.46
SGFNet (Ours)	45.2	128.5	28.3	82.0	0.64

4.4.2. Module-Wise Breakdown

Table 17 shows that SGM adds only 2.1 M parameters (4.7% of total) while contributing +1.7 pp mAP (45% of the total gain)—its gain-per-parameter ratio is 9.5× higher than the network average. The 28.3 FPS inference speed satisfies a 25 FPS real-time requirement for drone video processing. On NVIDIA Jetson AGX Orin, we achieve 15.2 FPS with TensorRT optimization.

Table 17. Computational cost breakdown by module.

Module	Params (M)	FLOPs (G)	Latency (ms)	% Total
Backbone (ResNet-50 × 2)	25.6	82.4	18.2	51.5%
FPN	8.3	24.1	5.3	15.0%
FDM	6.8	15.6	8.4	23.8%
SGM	2.1	3.2	1.8	5.1%
AGC	1.2	2.1	1.0	2.8%
Detection Head	1.2	1.1	0.6	1.8%
Total	45.2	128.5	35.3	100%

Failure Case Analysis. We systematically categorize failure cases and quantify their impact on the test set:

(1) Extreme Small Objects (<10 pixels, 8.3% of test set). Standard objects (>20 pixels) achieve 84.5% mAP, small objects (10–20 pixels) 76.8%, and tiny objects (<10 pixels) only 68.2%—a 16.3 pp drop. The primary cause is FDM’s fixed $3\times 3$ Laplacian kernel, which spans a significant portion of tiny objects and cannot resolve fine-grained frequency details. Mitigation: multi-scale Laplacian kernels ($3\times 3$, $5\times 5$, $7\times 7$) at lower pyramid levels could provide scale-adaptive frequency decomposition.

(2) Severe Modality Degradation (5.1% of test images). Normal conditions achieve 82.0% mAP, moderate degradation (50% quality) 79.3% (−2.7 pp), and severe degradation (10% quality) 71.2% (−10.8 pp). SGM cannot recover missing information because the P5 semantic mask degrades when both FPN streams carry unreliable features. Mitigation: a modality reliability estimation module that down-weights degraded modality features before FPN.

(3) Ambiguous Orientations (aspect ratio < 1.2, 12.7% of test set). These objects produce ΔAngle of 11.3° (vs. 5.1° for aspect ratio > 2.0) with AA@5° dropping from 95.2% to 68.4%. This stems from intrinsic geometric ambiguity: when width and height are nearly equal, orientation becomes ill-defined. No feature-level method can reliably distinguish 0° from 90°. Mitigation: orientation uncertainty estimation that explicitly models angular ambiguity for near-square objects.

5. Discussion

Semantic guidance and closed-loop feedback.Table 7 indicates that performance gains are driven by semantic guidance rather than backbone capacity. The baseline concatenation model reaches 78.2% mAP with the same backbone and detection head, implying that discriminative features are already present in the dual-stream representations; the key challenge is selecting task-relevant features for fusion. The +1.7 pp contribution of SGM (45% of total gain) confirms that explicit semantic feedback is more effective than gradient-only task awareness. By extracting importance masks from P5 features optimized by detection loss, SGM shifts fusion decisions from input-adaptive to task-driven criteria; detected object regions receive 10% higher fusion weights (measured ratio $1.098\pm 0.012$), consistent with the theoretical prediction $1+\lambda =1.10$ (Theorem 1). This trend aligns with application-oriented fusion [1,3], where semantic-level guidance outperforms pixel-level or feature-level criteria.

Module interactions and geometric alignment. Beyond individual contributions, Table 7 suggests interaction effects among the modules. Without FDM’s frequency separation ($\Phi$: 0.42 → 0.87), SGM adds only +0.9 pp on concatenated features, compared with +1.7 pp on frequency-separated representations. This suggests that SGM’s semantic masks distinguish object-relevant features from noise more effectively when features occupy orthogonal frequency channels. The orthogonality constraint ($\Phi \ge 0.85$) reduces cross-channel interference and amplifies semantic modulation. Table 7 also indicates that AGC improves angular precision more than mAP (+0.8 pp mAP but 13% ΔAngle reduction), reflecting geometric misalignment in standard convolutions: for a 3 × 3 kernel applied to a 45°-rotated vehicle, approximately 50% of sampling points fall on background pixels. AGC rotates the sampling grid and achieves >95% object coverage when $|\Delta \theta |<5°$, directly addressing this limitation. The per-category analysis (Table 6) shows larger gains for high-aspect-ratio objects (Freight Car: 4.0–6.0), which are more sensitive to receptive-field misalignment, consistent with ARC [23].

Limitations and future directions. Several limitations remain. First, Table 17 shows that the FDM Transformer component contributes 23.8% of the total latency; linear attention variants of the Transformer [30] may reduce overhead while preserving frequency separation quality. Second, evaluation is limited to a single dataset (DroneVehicle), currently the only publicly available benchmark providing both aligned RGB-IR pairs and oriented bounding box annotations at sufficient scale. Existing alternatives either lack thermal IR modality (DOTA [37]), use horizontal annotations (M³FD (Multi-scenario Multi-modality Benchmark Dataset) [11]), or provide insufficient training samples. Our scene-conditioned analysis (Table 4) partially addresses this by demonstrating consistent gains across three distinct illumination regimes. Additional RGB-IR oriented detection benchmarks would strengthen generalizability claims. Third, SGM cannot compensate for severe modality degradation (<10% quality, mAP drops to 71.2%), suggesting that incorporating modality reliability estimation is necessary for graceful degradation under adverse sensor conditions.

6. Conclusions

This paper presents SGFNet, a semantic-guided fusion network with closed-loop feedback in which P5 detection features modulate fusion weights via learned semantic masks. FDM enforces orthogonal cross-modal alignment ($\Phi \ge 0.85$), enabling SGM to enhance task-relevant features with validated weight amplification ($1+\lambda$), while AGC aligns receptive fields with predicted orientations for angular precision. Experimental results on DroneVehicle show that SGFNet achieves 82.0% mAP@0.5 (+3.2 pp over DMM) and 6.2° mean ΔAngle (16% reduction). Ablation studies validate the two-stage training strategy, the superiority of internal over external semantic guidance, and the effectiveness of Laplacian–Gaussian frequency decomposition. Semantic feedback provides the largest individual gain (+1.7 pp, 45% of total improvement), supporting the closed-loop fusion paradigm.

Based on the observed limitations, future work will: (1) extend the closed-loop paradigm to semantic segmentation and multi-object tracking; (2) replace the Transformer in FDM with linear attention mechanisms for real-time processing on embedded platforms; and (3) incorporate modality reliability estimation to support graceful degradation under adverse sensor conditions.