VODet: A Vertex Offset-Based Method for Oriented Object Detection in Remote Sensing Images

Highlights

What are the main findings?

• We propose VODet, a novel vertex offset-based oriented object detection framework that integrates a coupled vertex offset representation, a lightweight direction classifier, a hierarchical special attention model (HSAM), and a task-specific matching rule with MPDIoU loss.
• Our method achieves competitive performance on three widely used remote sensing benchmarks (DOTA-v1.0, HRSC2016, and UCAS-AOD), with particular advantages for objects of large aspect ratios and dense arrangements.

What are the implications of the main findings?

• The proposed coupled vertex offset paradigm offers a new perspective for oriented object detection by fundamentally avoiding the boundary discontinuity problem inherent in angle-based representations.
• This framework provides a practical and generalizable solution for high-precision detection of arbitrarily oriented objects, supporting applications such as urban planning, maritime surveillance, and disaster monitoring.

Abstract

To tackle the challenges in oriented object detection, such as discontinuous angle regression, low precision in discrete classification, and high complexity in probabilistic modeling, this paper proposes VODet (Vertex Offset-based Detector), a novel oriented object detection method. VODet models the geometric structure of rotated objects through a coupled vertex offset mechanism, inherently avoiding boundary discontinuity. Specifically, it first regresses the horizontal enclosing rectangle of a target from horizontal anchors, then predicts only two offset ratios to reconstruct the four vertices of the rotated bounding box. For near-horizontal critical cases, a lightweight direction classifier adaptively distinguishes horizontal boxes from rotated ones, reducing regression complexity and improving robustness. To further enhance feature representation, we design a hierarchical special attention model (HSAM) based on Swin Transformer blocks, which effectively captures both local details and global contextual dependencies. Extensive experiments on DOTA, HRSC2016, and UCAS-AOD demonstrate that VODet achieves competitive detection accuracy (80.44% mAP on DOTA, 90.42% AP on HRSC2016, and 90.11% mAP on UCAS-AOD), with particular advantages for objects of large aspect ratios and dense arrangements. Moreover, VODet achieves the highest inference speed (26.7 FPS) and the lowest computational complexity (192.3 GFLOPs) among the compared methods, demonstrating a favorable accuracy-efficiency trade-off. These results confirm the effectiveness and practical value of the proposed method.

1. Introduction

Satellite and aerial remote sensing advances have promoted their widespread use in urban planning, resource exploration, environmental monitoring, and national defense security [1,2,3]. Efficient and accurate object extraction from remote sensing imagery remains a central challenge in this field. Traditional detectors are mostly designed for horizontal bounding boxes (HBBs) [4,5,6]. Targets in remote sensing imagery, however, are often characterized by arbitrary orientations, dense arrangements, and large aspect ratios [7,8]. In such cases, HBB-based methods are prone to incorrectly suppressing valid detection boxes during the NMS process due to excessively high IoU between neighboring boxes, which results in missed detections.

To address these limitations, numerous oriented object detection methods have been developed, which construct oriented bounding boxes (OBBs) by introducing a rotation angle [9,10,11]. The five-parameter representation (x, y, w, h, θ) is commonly adopted, where (x, y) is the center, w and h the dimensions, and θ the rotation. Although such approaches have achieved improvements in certain scenarios, the periodic nature of the rotation angle leads to an inherent contradiction in the regression process. Taking the widely used long-side definition method as an example [12], the angle θ belongs to [−π/2, π/2). When the target angle approaches the critical point of the definition domain, minor geometric differences can cause a sudden jump in the angle value [7,13], thereby hindering smooth regression losses from accurately capturing the geometric similarity between prediction and ground truth. This issue is particularly pronounced in the detection of targets with large aspect ratios, severely limiting detection accuracy. Figure 1 shows the detection results of a large truck under boundary conditions using the five-parameter method. It can be observed that among the 11 predicted boxes whose ground-truth angles are close to −π/2, 7 boxes exhibit significant deviations in angle prediction. This issue arises because the angular parameter θ is defined within the periodic interval [−π/2, π/2). When the ground-truth angle of a target approaches this domain boundary, directly applying Smooth L1 loss for regression can cause the angle value to undergo a periodic jump due to minor angular variations—for example, leaping from −π/2 to π/2. This results in a discontinuous and drastic change in the loss value. Such discontinuity in the loss function severely impedes the network’s ability to effectively learn targets with boundary angles, i.e., those where θ ≈ ±π/2.

Existing approaches to address angle discontinuity can be grouped into three categories, each with distinct limitations. First, converting angle prediction into a classification task [13,14,15] mitigates boundary jumps but suffers from limited precision due to discretization, making it difficult to meet the stringent accuracy requirements for large aspect ratio targets. Moreover, introducing a classification head significantly increases model complexity. Second, methods based on probability distributions [16,17,18] represent the rotated box as a 2D Gaussian distribution, transforming the bounding box matching problem into a distribution similarity measure, thereby better circumventing angle discontinuity. However, computing distribution similarity has high complexity, and the loss function lacks intuitive geometric interpretability. Third, feature design-based methods [19,20,21,22] aim to achieve rotation invariance by enhancing feature representation and decoupling it from target orientation via spatial transformation and alignment. While these methods improve rotation modeling capability, they significantly increase network depth, parameter count, and computational cost, reducing efficiency. Recently, methods based on vertex sliding or vertex offset mechanisms [23,24,25], which define orientation by directly modeling the geometric vertices of the target, have shown considerable potential in oriented object detection by better avoiding the periodicity issues of angle regression. While existing vertex offset methods have alleviated angular periodicity, they typically suffer from three major limitations: (1) geometric constraint violations among multiple independently regressed vertices, (2) increased regression complexity for near-horizontal objects, and (3) insufficient multi-scale feature representation for small and densely arranged targets.

To overcome these limitations, this paper proposes VODet, a vertex offset-based framework that introduces a novel coupled vertex offset representation, a lightweight direction classifier specifically designed for near-horizontal critical cases, an enhanced backbone integrating the proposed hierarchical special attention model (HSAM) with FPN, and a task-specific matching rule with Minimum Points Distance Intersection over Union (MPDIoU) loss. This integrated design achieves both high precision and robust detection performance. In our experiments, VODet achieves the highest accuracy among single-stage methods and the best efficiency among all compared methods, offering a favorable accuracy-efficiency trade-off. The key contributions of this paper are:

• We propose a continuous coupled vertex offset representation, which regresses only two offset ratios relative to the horizontal enclosing box to reconstruct the rotated bounding box, thereby enforcing geometric consistency and avoiding boundary discontinuity.
• We introduce a lightweight direction classifier that dynamically distinguishes horizontal from rotated boxes in near-horizontal cases, thereby converting challenging regression into stable classification and improving robustness with minimal cost.
• We design an enhanced backbone with the proposed HSAM and FPN. Built on ResNet-101, it strengthens key region representation and multi-scale features, thereby leading to better detection of small and densely arranged objects.
• We develop a task-specific matching rule with size and position constraints for high-quality anchor assignment and adopt MPDIoU loss to minimize corner distances, thereby producing more accurate horizontal boxes and boosting overall localization accuracy.

In addition to the above detection methods, the quality of remote sensing images is often degraded by adverse imaging conditions such as haze, cloud cover, and nighttime low-light environments. Recent advances in image restoration techniques, including the Bayesian-based fractional-order variational model for nighttime dehazing [26], the NGSCD model for glow suppression [27], as well as IHDCP [28] and VNDHR [29] for single-image dehazing, have shown promise in recovering high-quality images from degraded inputs. In this work, we focus on the detection problem under standard imaging conditions, while considering that robustness to degraded conditions is an important direction for future investigation.

Early rotated object detectors were built upon horizontal detection frameworks by incorporating an additional rotation angle θ, yielding the five-parameter representation (x, y, w, h, θ). As an example, RRPN [10] designed rotated anchors with angular priors, treating the angular difference between anchors and target boxes as a regression target and optimizing the angle using smooth-L1 loss. SCRDet [7] also regressed angular offsets based on five parameters, employing a smooth L1 loss function that incorporates an IoU constant factor to alleviate the boundary discontinuity caused by angle periodicity. R³Det [8] further introduced an approximate SkewIoU loss to address the challenge of angle periodicity, combined with a feature refinement module for progressive optimization of angle parameters. Although these methods have alleviated the periodic jump phenomenon, they have not fundamentally resolved the discontinuity and instability inherent in angular representation.

Typical methods based on angle classification include CSL [13] and DCL [12]. CSL divides the angle into discrete categories and introduces a circular smooth label for each angle to mitigate semantic jumps between adjacent angle categories. DCL proposes a densely coded label, representing angles with shorter code lengths using binary or Gray codes to reduce model parameters. However, discrete angle prediction suffers from coarse discretization, hindering its ability to meet the precision required for large aspect ratio targets (e.g., ships and trucks). Furthermore, converting continuous rotation angle prediction into a discrete classification task significantly increases model parameters and computational complexity, leading to reduced training efficiency.

Methods based on probability distribution transform rotated boxes into 2D Gaussian distributions, optimizing bounding box parameters indirectly by measuring distribution differences. For instance, GWD Loss [16] employs the Wasserstein distance to quantify the discrepancy of two Gaussian distributions, while KLD Loss [17] adopts KL divergence as the regression loss and introduces a gradient coupling scheme that adaptively adjusts the optimization process according to the target’s aspect ratio, improving detection accuracy for angle-sensitive targets. However, the loss computation in these methods involves complex matrix operations. Additionally, approximating rectangular bounding boxes with Gaussian distributions introduces inherent geometric representation errors, limiting localization accuracy and efficiency.

In feature design-based methods, RoI Transformer [19] first converts horizontal region proposals into rotated regions, then performs rotation transformation and alignment on the feature map based on these rotated regions to extract rotation-invariant features independent of target orientation. ReDet [20] starts from the feature level, extracting rotation-equivariant features through equivariant convolution and enhancing orientation robustness with rotation-invariant RoI Align. While these methods improve rotation modeling, they significantly increase network depth and parameters, slowing inference and limiting real-time applicability.

The vertex offset mechanism has recently gained attention for directly modeling target vertex geometry, thereby circumventing the inherent drawbacks of angle parameterization. Oriented R-CNN [24] uses horizontal anchors as a reference, generating parallelogram candidate boxes by predicting the offsets (∆α, ∆β) of the center point and the enclosing box’s position and size (x, y, w, h), which are then decoded into rotated rectangles. However, numerical errors may be introduced during the decoding from parallelogram to rotated rectangle, causing box deformation. Moreover, it does not consider boundary issues in angle regression, leading to prediction ambiguity when the target orientation is near the angle boundary, affecting the detection stability of strongly directional targets like ships and vehicles.

Gliding Vertex [23] uses the horizontal enclosing rectangle as a reference box, regressing the sliding ratios of vertices on its four edges to reconstruct the rotated box through geometric constraints. However, this method requires independently predicting four offset coefficients, which can violate spatial coupling constraints and lead to distorted predicted boxes. Additionally, the nonlinear relationship between sliding ratios and pixel coordinates increases regression difficulty, especially for endpoints of long-edge targets like runways.

BBAVectors [25] adopts a vector-based vertex offset mechanism for rotated object detection. It detects the object’s center point and regresses boundary-aware vectors emanating from this center toward the four edges, with vertex coordinates ultimately synthesized from these vectors. However, this method heavily relies on the initial center point localization accuracy and requires regressing multiple vectors and external size parameters for each target, making the regression targets relatively complex. In dense or cluttered scenes, this increases the model’s learning and convergence difficulty.

Gliding Vertex [23] regresses four independent sliding ratios, which may violate geometric constraints. In contrast, VODet regresses only two coupled offsets, enforcing geometric consistency. BBAVectors [25] regresses multiple boundary-aware vectors from the center point, increasing regression complexity, while VODet directly regresses vertex offsets from the HBB corners. Oriented R-CNN [24] decodes parallelogram proposals into rotated boxes, which may introduce numerical errors and extra costs, whereas VODet directly reconstructs OBB vertices via a deterministic formulation. In addition, VODet introduces a Direction Classifier for near-horizontal cases and an HSAM for enhanced feature representation. These components, together with the coupled offset formulation, distinguish VODet from prior vertex-based methods.

Although existing research validates the effectiveness of the vertex offset mechanism, three major challenges remain:

• The geometric constraints between multiple independently regressed offsets are often neglected, resulting in distorted box predictions and boundary discontinuity.
• The regression complexity increases significantly for near-horizontal objects, where offset values become extremal and difficult to predict accurately.
• Most existing methods lack sufficient multi-scale feature representation, limiting their performance on small and densely arranged targets.

To address these limitations, we propose VODet, which introduces a coupled vertex offset representation, a lightweight direction classifier, an enhanced backbone with attention and FPN, and a task-specific matching rule with MPDIoU loss for precise and robust oriented object detection.

2. Materials and Methods

2.1. Oriented Object Detection Network

As illustrated in Figure 2, VODet adopts ResNet-101 as its backbone, with all weights from conv1 to conv5_x retained. To strengthen feature modeling, we integrate the proposed HSAM after the C5 (output of conv5_x). HSAM enhances the information representation of key regions and suppresses background noise in the feature map through a hierarchical window-based self-attention mechanism. A detailed description of HSAM is provided in Section 3.2.

Subsequently, a set of pyramidal features is constructed utilizing the FPN [30] architecture. First, a 1 × 1 convolution is applied to the output of HSAM to generate P5. Subsequently, P5 is upsampled and added to C4, which has been reduced in dimension via a 1 × 1 convolution, after which a 3 × 3 convolution is applied to produce P4. Similarly, P4 is upsampled again and added to the dimension-reduced C3, then smoothed to obtain P3.

The model performs detection on three feature pyramid levels: {P3, P4, P5}. The detection head at each level consists of 1 × 1 convolutional layers. We adopt a vertex offset regression strategy based on horizontal anchors. At each spatial position on each level’s feature map, three horizontally oriented anchors of aspect ratios {1:2, 1:1, 2:1} are assigned. The anchor sizes are 16, 32, and 64, respectively, when operating on 640 × 640 pixel input images. If other input sizes are used, the anchor sizes need to be scaled proportionally. A given anchor is encoded as a vector a = (a_x, a_y, a_w, a_h), consisting of its center coordinates (a_x, a_y) and its dimensions a_w (width) and a_h (height).

2.2. Hierarchical Special Attention Model

Figure 3 illustrates the overall architecture of the HSAM. The detailed implementation process of HSAM is described as follows.

The module receives an input feature map X ∈ R^B×C×H×W. A 1 × 1 convolutional layer is first applied to expand the channel dimension from C to 2C. The expanded feature map is then split along the channel dimension into two parallel branches. One branch serves as an identity shortcut branch, which directly preserves the original feature information without any additional transformation. The other branch acts as an attention enhancement branch and is fed into two successive Swin Transformer blocks [31] for feature modeling.

Since the Swin Transformer Block expects the [B, H, W, C] format, whereas our module uses [B, C, H, W], a dimension transformation is applied before the attention branch enters the first block to convert [B, C, H, W] to [B, H, W, C]. After sequentially passing through the two stacked Swin Transformer Blocks, an inverse transformation restores the format back to [B, C, H, W] to maintain dimensional consistency with the identity shortcut branch.

Finally, the features processed by the attention enhancement branch and the features from the identity shortcut branch are concatenated along the channel dimension. A 1 × 1 convolutional layer is then applied to fuse the concatenated features, producing an output feature map Y ∈ R^B×C×H×W with the same dimensions as the input. In this way, HSAM enhances and optimizes the feature representation, providing more discriminative features for subsequent detection tasks. The entire HSAM can be formulated as

Y = Conv_1×1 ([STB (Permute(X1))||X2])

(1)

where X1 and X2 denote the two split branches, Permute(.) represents the dimension transformation operations, STB(.) denotes the two stacked Swin Transformer Blocks, and || denotes channel-wise concatenation.

2.3. Network Outputs

The network output consists of five parts. The first part comprises the regression parameters for the horizontal predicted box, outputting the offset and scaling factors with respect to the anchor, represented by the vector δ = (δ_x, δ_y, δ_w, δ_h), where (δ_x, δ_y) represents the center point offset, and (δ_w, δ_h) represents the scaling factors for width and height. The second part is the vertex offset parameters, outputting the offset ratios of the oriented target box’s topmost and rightmost vertices with respect to the horizontal target box’s top-right vertex, represented by the vector d = (d_t, d_r). The third part is the confidence score c of the predicted box, used to distinguish foreground from background objects. The fourth part is the direction category o of the predicted box, indicating whether it is a horizontal box or an oriented box (see Section 2.4). The final part is a set of classification probabilities k = (k₁, k₂, …, k_n), with n representing the total number of target categories. All network outputs are processed by the sigmoid function to ensure values in the range (0, 1).

By decoding the outputs δ and d, the horizontal predicted box (c_x, c_y, c_w, c_h) and the vertex offset information (d_w, d_h) for the rotated predicted box with respect to the horizontal predicted box can be obtained. The decoding process is shown in Equation (2).

$$\left\{\begin{array}{l}{c}_w=a_w\times {(2{\delta }_w)}^2, c_h=a_h\times {(2{\delta }_h)}^2\\ c_x=2{\delta }_x-0.5+a_x, c_y=2{\delta }_y-0.5+a_y\\ d_w=d_t\times c_w, d_h=d_r\times c_h\end{array}\right.$$

(2)

As shown in Figure 4, (c_x, c_y) and (c_w, c_h) denote the center and dimensions (width and height) of the horizontal predicted box, respectively. d_w and d_h represent the offsets of the oriented target box’s topmost and rightmost vertices with respect to the top-right corner of the horizontal predicted box.

Based on the horizontal predicted box information and vertex offsets, the coordinates of the four vertices (p₁, p₂, p₃, p₄) of the oriented target box can be calculated using the vertex offset representation method, see Equation (3):

$$\left\{\begin{array}{l}{p}_1(x, y)=(c_x+c_w/2-d_w, c_y-c_h/2)\\ p_2(x, y)=(c_x+c_w/2, c_y-c_h/2+d_h)\\ p_3(x, y)=(c_x-c_w/2+d_w, c_y+c_h/2)\\ p_4(x, y)=(c_x-c_w/2, c_y+c_h/2-d_h)\end{array}\right.$$

(3)

The proposed vertex offset representation exhibits several desirable theoretical properties. As illustrated in Figure 4, the blue rectangle h₁h₂h₃h₄ denotes the OBB’s corresponding HBB. In our method, the HBB and the vertex offset vectors are regressed through δ and d, respectively. Since the vertex offset vectors are fixed, the coordinates of points p₁ and p₂ are uniquely determined. Let the length of p₁p₂ be l, and the angle ∠h₂p₁p₂ be α. It follows that ∠h₄p₃p₄ = α. Consequently, the length of p₄h₄ is l×sinα, and the length of h₄p₃ is l*cosα. These two lengths uniquely determine the coordinates of p₃ and p₄. Therefore, the entire OBB p₁p₂p₃p₄ is uniquely determined. Furthermore, unlike angle-based representations that suffer from sudden jumps at domain boundaries, the vertex offsets d_t and d_r vary continuously with the target’s rotation, ensuring stable gradient propagation during training. The offsets also provide intuitive geometric interpretability, as they directly encode the spatial positions of the OBB’s topmost and rightmost vertices relative to the HBB’s top-right corner.

2.4. Directionality

The network distinguishes the directionality of oriented targets using the vertex offset features d(d_t, d_r). Generally, targets with similar rotation angles have similar vertex offset features. However, when an oriented target is nearly horizontal, the offset feature d becomes ineffective in representing the actual differences between oriented targets. As shown in Figure 5, the two red-oriented targets have the same dimensions but different rotation angles. Figure 5a shows a target rotated 3.5° clockwise around its center from a horizontal box, while Figure 5b shows one rotated 3.5° counterclockwise. Although their rotation angles differ only slightly and they are visually very similar, their offset features differ significantly. For Figure 5a, the offset feature is d₁ = (0.96, 0.08); for Figure 5b, d₂ = (0.04, 0.93). This feature variation phenomenon presents a dual challenge for the network when handling oriented targets close to the horizontal state: first, the offset feature d becomes ineffective in representing the actual differences between targets; second, the elements in d tend towards 0 or 1, significantly increasing the regression difficulty. We define such oriented boxes close to the horizontal state as critical cases, a special scenario the model needs to handle.

To effectively handle critical cases, we process OBBs as one of two types: HBBs or RBBs (Rotated Bounding Boxes). Here, RBBs represent all standard rotated objects, while critical cases are specifically converted into HBBs. This conversion eliminates orientation ambiguity, providing the network with more stable regression targets and thus facilitating better convergence. When the network subsequently encounters such an HBB (originally a critical case), the direction category o and the regression vector δ enable the reconstruction of an accurate OBB. The ground truth label for o is determined by Equation (4):

$$o=\left\{\begin{array}{l}1(\mathrm{RBB}) R_{area}(\mathrm{OBB},\mathrm{HBB})<o_t\hfill \\ 0(\mathrm{HBB}) \mathrm{otherwise},\hfill \end{array}\right.$$

(4)

where R_area is the area ratio of the OBB to its HBB, and o_t represents the Rarea threshold. The detailed decoding process, including the handling of such critical cases, is systematically illustrated in Algorithm 1.

Algorithm 1: Vertex Offset Decoding and Direction- Aware Bounding Box Construction

Input:

Ground truth boxes G = {g1, g2, …, gm} for current batch;

Anchor boxes A = {A1, A2, …, AL} for each pyramid level;

Pyramid levels L = {P3, P4, P5};

Hyperparameters: Rarea threshold ot, size constraint threshold rt;

Output:

Oriented bounding boxes B and total loss Ltotal;

Initialize positive sample set P ← ∅, bounding box set B ← ∅;

for each ground truth box g ∈ G do

Compute ground truth direction label ogt using Equation (4) with threshold ot;

Find anchors in A satisfying rmax < rt and position constraints ();

Add matching anchors to positive sample set P;

end

for each level l ∈ L do

for each anchor a ∈ Al do

Decode horizontal box (cx, cy, cw, ch) from δ using Equation (2);

Decode vertex offsets (dt, dr) ← σ(d);

Decode direction category oˆ ← ⌊σ(o) + 0.5⌋;

if oˆ = 1 then

Compute vertices V from (cx, cy, cw, ch) and (dt, dr) using Equation (3);

else

Set vertices V as corners of (cx, cy, cw, ch);

end

Add V to bounding box set B;

end

end

Compute total loss Ltotal using Equation (5);

return B, Ltotal;

2.5. Discussion on Special Cases

The theoretical analysis in Section 2.3 assumes general cases. We further discuss several special cases to demonstrate the robustness of the proposed vertex offset representation:

• Arbitrary rotation angles. For targets rotated across the angle boundary (e.g., near ±π/2), the HBB remains well-defined, and the vertex offsets (d_t, d_r) remain continuous with respect to the target’s orientation. This avoids the sudden loss discontinuity that occurs in angle-based methods.
• Near-square targets. When a target is approximately square, the geometric derivation in Section 2.3 still holds, as sinα and cosα remain well-defined. The uniqueness of the OBB is preserved.
• Extreme aspect ratio targets. For highly elongated targets (e.g., ships, bridges), the geometric derivation still holds regardless of the aspect ratio. The coupled vertex offset representation naturally accommodates such targets, as the HBB adapts to the target’s extent, and the offsets encode the relative positions of the intersection points.
• Vertex ordering. The coupled offset representation enforces a fixed vertex ordering: p₁ is the intersection on the top edge, p₂ on the right edge, p₃ on the bottom edge, and p₄ on the left edge of the HBB. This avoids the vertex sorting ambiguity that may arise in methods that regress independent vertices, as the geometric constraints are inherently encoded in the coupled formulation.

2.6. Loss Function

The loss function of VODet is presented in Equation (5), which consists of five terms: L_cls, L_obj, L_loc, L_off and L_ort for classification, confidence, bounding box regression, vertex offset, and directionality, respectively.

Loss = λ₁L_cls + λ₂L_obj + λ₃L_loc + λ₄L_off + λ₅L_ort

(5)

Here, L_cls and L_ort are binary cross-entropy losses, calculated only for positive samples. L_obj is also a binary cross-entropy loss, calculated for all samples. L_off uses the smooth L1 loss, calculated for positive samples.

L_loc represents the discrepancy between the predicted horizontal box and its ground truth counterpart. This paper uses MPDIoU [32] for its calculation. It works for box regression scenarios with or without overlap between boxes, addresses the failure of traditional IoU variants like GIoU/CIoU when aspect ratios are identical but sizes differ, simplifies the computation process, improves object detection accuracy and training convergence speed, and shows enhanced stability in challenging scenes. The formula for MPDIoU is:

$$\mathrm{MPDIoU}={\displaystyle \frac{A\cap B}{A\cup B}}-{\displaystyle \frac{l_1^2+l_2^2}{w^2+h^2}}$$

(6)

where A corresponds to the ground truth box and B to the predicted box, while w and h represent the image’s width and height, and l₁ and l₂ are given by Equation (7).

$$\left\{\begin{array}{l}{l}_1^2={(x_1^B-x_1^A)}^2+{(y_1^B-y_1^A)}^2\\ l_2^2={(x_2^B-x_2^A)}^2+{(y_2^B-y_2^A)}^2\end{array}\right.$$

(7)

In Equation (7), the top-left and bottom-right corners of A are (x₁^A, y₁^A) and (x₂^A, y₂^A), and those of B are (x₁^B, y₁^B) and (x₂^B, y₂^B), respectively. The complete loss computation follows the procedure outlined in Algorithm 1.

2.7. Positive Sample Matching Rule

The model regresses oriented objects from a set of predefined anchors. The two criteria listed below must be met for an anchor to be labeled positive:

• Size Constraint. Given a ground truth box and an anchor, let r_w and r_h be the ratios of their widths and heights, respectively. We compute r_max = max(r_w, 1/r_w, r_h, 1/r_h). The size constraint is satisfied if r_max < r_t, where r_t is a threshold (set to 2 in this work). This guarantees that they have comparable aspect ratios and scales.
• Position Constraint. The ground truth box is projected onto the corresponding prediction feature layer. The assignment of positive samples is based on the spatial relationship between the center points of the ground truth box and the anchor. If the distance between an anchor’s center point (a_x, a_y) and a ground truth box’s center point (c_x*, c_y*) satisfies Equation (8), then they satisfy the positional relationship.

c_x* − a_x ∈ (−0.5, 1.5), c_y* − a_y ∈ (−0.5, 1.5)

(8)

The overall pipeline, including positive sample assignment, vertex offset decoding, direction-aware box construction, and multi-task loss computation, is summarized in Algorithm 1.

2.8. Datasets

The DOTA-v1.0 is a widely used benchmark for oriented object detection in remote sensing. It provides 2806 large-scale images (800 × 800 to 4000 × 4000 pixels) with 188,282 instances spanning 15 categories. These include plane (PL), baseball diamond (BD), bridge (BR), ground field track (GTF), small vehicle (SV), large vehicle (LV), ship (SH), tennis court (TC), basketball court (BC), storage tank (ST), soccer-ball field (SBF), roundabout (RA), harbor (HA), swimming pool (SP), and helicopter (HC). The training, validation, and test sets in DOTA-v1.0 are split in a 1/2, 1/6, 1/3 ratio, respectively. For our experiments, all training and validation images were split into 1024 × 1024-pixel patches with a 200-pixel overlap before training. Images with a width or height less than 1024 pixels were padded to 1024 pixels.

The HRSC2016 dataset contains imagery collected from six well-known harbors, containing both nearshore and offshore scenes, with a total of 1061 ship images. The target category is ships only. The images are divided into 436 for training, 181 for validation, and 444 for testing. Image dimensions range from 300 × 300 pixels to 1500 × 900 pixels. For our experiments, all training and validation data were uniformly cropped to 512 × 512 pixels with a 150-pixel overlap.

UCAS-AOD comprises 1510 images, containing 14,596 annotated instances across two categories: airplanes and cars. The image size is 659 × 1280 pixels. For the experiments, we performed a random split of the dataset into 1110 training and 400 test images.

3. Results

To validate the performance of our approach, we evaluated it on three public benchmarks for oriented object detection: DOTA-v1.0 [33], HRSC2016 [34], and UCAS-AOD [35]. The following sections detail these datasets, experimental settings, and results.

3.1. Experimental Details

Experiments were conducted on hardware equipped with an NVIDIA GeForce RTX 3090 GPU and a PyTorch-1.8.1 software environment. The model uses the Adam optimizer [36] with initial learning rate, momentum, and weight decay set to 1 × 10⁻³, 0.9, and 1 × 10⁻⁴, respectively. The batch size is 8. The training epochs for each dataset are: DOTA-v1.0, 80; HRSC2016, 100; UCAS-AOD, 100. The learning rate is reduced to 1 × 10⁻⁵ after 50 epochs for each dataset. Data augmentation includes random rotation, horizontal flip, vertical flip, and Mosaic. The R_area threshold o_t in Equation (4) is set to 0.90 by default. The loss weight parameters in Equation (5) are: ${\lambda }_1$ = 1, ${\lambda }_2$ = 1, ${\lambda }_3$ = 2, ${\lambda }_4$ = 1.5, ${\lambda }_5$ = 0.5.

3.2. Ablation Studies

3.2.1. Validation of Module Effectiveness

We perform ablation studies using DOTA-v1.0 to validate the efficacy of VODet. The baseline model, denoted as 5PDet, employs the (x, y, w, h, θ) representation for angle prediction and uses ResNet-101 with FPN as the backbone. In 5PDet, the vertex offset channel in the detection head is replaced by an angle regression channel. The angle θ follows the long-side representation, where θ ∈ [−π/2, π/2). To increase prediction accuracy, the angle channel does not directly regress θ but encodes it. Assuming the output of the angle prediction branch is p_θ, the predicted angle $\widehat{\theta }$ is calculated using the following formula:

$$\widehat{\theta }=(\sigma (p_{\theta })-0.5)\ast \pi$$

(9)

where σ is the sigmoid function. The purpose of encoding θ is to leverage the sigmoid function’s output range of (0, 1), preventing the network’s predicted angle from exceeding the definition range of the long-side representation and reducing regression difficulty.

The baseline model employs CIoU loss [37] for the bounding box loss L_loc. During ablation experiments, we evaluate each module both independently and in combination to validate their individual contributions and interaction effects. As shown in

Table 1. Ablation experiment results of different VODet module combinations on the DOTA-v1.0 dataset.

Configuration	HSAM	Vertex Offset	Direction Classification	MPDIoU	mAP (%)
5PDet (Baseline)	×	×	×	×	70.23
5PDet	√	×	×	×	72.17
5PDet	×	×	√	×	72.69
5PDet	×	×	×	√	71.14
VODet	×	√	×	×	73.58
VODet	√	√	×	×	76.56
VODet	√	√	√	×	78.92
VODet	√	√	√	√	80.44

, adding HSAM alone improves mAP from 70.23% to 72.17% (+1.94%), demonstrating its effectiveness in feature enhancement. Direction classification alone raises mAP to 72.69% (+2.46%), confirming its ability to handle critical cases. MPDIoU alone yields a modest improvement to 71.14% (+0.91%), indicating that its benefit is more pronounced when combined with other modules. Vertex offset alone achieves 73.58% (+3.35%), proving the effectiveness of the vertex offset mechanism even without HSAM.

When HSAM and Vertex Offset are used together, mAP jumps to 76.56%, a substantial improvement over either module alone. This synergy suggests that HSAM-enhanced features provide a more reliable basis for vertex regression. Adding direction classification further improves mAP to 78.92% (+2.36%), indicating that it addresses a distinct challenge—critical cases—that is complementary to vertex offset regression. Finally, incorporating MPDIoU loss brings the performance to 80.44% (+1.52%), demonstrating that refining horizontal box localization directly benefits the vertex reconstruction process.

Overall, the cumulative improvement from 70.23% to 80.44% (+10.21%) confirms that all modules contribute positively with minimal negative interactions, validating the compatibility of our design choices.

3.2.2. Effectiveness of the HSAM

Figure 6 visualizes the feature heatmaps of the baseline model without and with the proposed HSAM. Before HSAM is applied, the model exhibits scattered and relatively weak activations, making it difficult to fully capture the target regions, especially for small and densely arranged objects. After HSAM is integrated, the activated regions become more concentrated on the targets, and the response intensity is enhanced. As shown in the figure, for challenging categories such as ships (SH), HSAM enables the model to better distinguish foreground objects from complex backgrounds. Furthermore, for large-aspect-ratio objects like large vehicles (LV) and harbors (HA), HSAM produces more complete and continuous activations along the elongated target structures. These qualitative results demonstrate that HSAM effectively captures both local details and global contextual dependencies, leading to more discriminative feature representations and improved detection performance.

We further compare HSAM with several popular attention modules, including Squeeze-and-Excitation (SE) [38], Convolutional Block Attention Module (CBAM) [39], and Coordinate Attention (CA) [40], under the same baseline configuration. As presented in

Table 2. Comparison of different attention modules on the DOTA-v1.0 dataset.

Attention Module	mAP (%)	ΔmAP (%)
Baseline	70.23
SE [38]	70.35	0.12
CBAM [39]	70.18	−0.05
CA [40]	70.72	0.49
HSAM (ours)	72.17	1.94

, HSAM achieves the highest ΔmAP of 1.94%. This superior performance can be attributed to HSAM’s channel-split parallel architecture combined with Swin Transformer blocks, which capture both local details and global dependencies more effectively than other attention mechanisms.

3.2.3. Effectiveness of the Vertex Offset Mechanism

Figure 7 illustrates the visual comparison between 5PDet and VODet (using only vertex offset). Due to the discontinuity issue in rotation angles, 5PDet faces increased regression difficulty, leading to less accurate angle predictions. In Figure 7a–c, 5PDet can locate targets like planes, ships, and storage tanks, but the target orientations have significant deviations, especially for targets with larger aspect ratios, such as the harbors in Figure 7b. In contrast, VODet, using the vertex offset mechanism, overcomes the discontinuity problem of angle regression, enabling more accurate prediction of bounding box size and rotation angle, thus more tightly localizing oriented targets and improving detection accuracy in remote sensing images. The advantages of VODet are especially evident for slender objects like the harbors and dense scenes like the ships in Figure 7e.

3.2.4. Effectiveness of the Direction Classification Mechanism

Figure 8 compares the detection performance of the basic VODet and the VODet with the direction classification mechanism added. When the basic VODet model processes nearly horizontally distributed targets, such as the large vehicles in Figure 8a,b or the tennis court in Figure 8c, it often fails to accurately detect the nearly horizontal bounding boxes—these are the “critical cases” mentioned earlier. The reason for detection failure is that when a rotated target approaches a horizontal or vertical orientation, the elements of the vertex offset component d = (d_t, d_r), d_r tend towards 0 or 1, which lies at the boundary of the sigmoid function, increasing regression difficulty. To address this, we classify rotated bounding boxes into RBBs and HBBs, achieving accurate prediction for critical cases by predicting the direction category o. As shown in Figure 8, after adding the direction category, the model performs excellently in critical cases, accurately capturing rotated bounding boxes. Table 1 shows that adding direction classification improves VODet’s mAP by 2.36%.

The direction classifier relies on threshold o_t to distinguish critical near-horizontal cases (converted to HBBs) from standard rotated objects (remaining as RBBs). To investigate the sensitivity of this hyperparameter, we conduct ablation experiments by varying o_t from 0.75 to 0.95 with a step of 0.05, while keeping all other settings identical. As shown in

Table 3. Sensitivity analysis of o_t.

ot	0.75	0.80	0.85	0.90	0.95
mAP(%)	77.46	77.83	78.42	78.92	78.35

, the detection performance remains relatively stable across different o_t values, with mAP ranging from 77.46% to 78.92%. The best performance is achieved at o_t = 0.90 (78.92%), which is therefore chosen as the default threshold in our experiments. When o_t is too low (e.g., 0.75), more cases are misclassified as critical, leading to loss of orientation information, resulting in a lower mAP of 77.46%. When o_t is too high (e.g., 0.95), fewer cases receive the directional classification benefit, causing a slight performance drop to 78.35%. Overall, VODet exhibits low sensitivity to the choice of o_t within a reasonable range, demonstrating its robustness.

3.2.5. Effectiveness of the MPDIoU Loss Function

Figure 9 shows detection comparison results for VODet with and without the MPDIoU loss. Figure 9a–c show the oriented object detection results of VODet using CIoU. Figure 9d–f show the results of VODet using MPDIoU. It can be observed that VODet, guided by MPDIoU, achieves more accurate localization of targets. The bounding boxes predicted by VODet without MPDIoU have slight deviations from the ground truth. This improvement stems from MPDIoU’s ability to enhance the accuracy of horizontal box estimation for oriented targets through minimizing the corner point distances between predictions and their corresponding ground truths, which in turn boosts the localization precision of the final rotated boxes.

To further demonstrate the advantage of MPDIoU over CIoU for bounding box regression, we plot the training loss curves of the localization branch using CIoU and MPDIoU, respectively, in Figure 10. It can be seen that the model with MPDIoU converges faster and achieves a lower final loss compared to CIoU. This demonstrates that MPDIoU provides more effective gradient guidance during training, leading to more accurate bounding box regression.

3.2.6. Feature Map Visualization Interpretation

To highlight the advantages of VODet, we visualize the feature heatmaps of the baseline and our method in Figure 11. We select six instances of categories: PL, SV, LV, SH, TC, and HA, among which LV and HA represent large-aspect-ratio objects, while SV, LV, and SH are densely arranged objects. The figure shows that the baseline method fails to focus well on the objects due to angle regression discontinuity. In contrast, our method, which employs the vertex offset mechanism, orientation classification strategy, and MPDIoU loss, directs more attention to the objects themselves and enables precise localization. It not only effectively distinguishes between background and foreground but also excels in extracting features from large-aspect-ratio and densely arranged objects.

4. Discussion

4.1. Comparison with State-of-the-Art Methods

The reported results of the compared methods are collected from their original publications, where experimental settings (e.g., input sizes, training strategies, data augmentation, and post-processing) may vary. To ensure a fair comparison, we select methods that use the same backbone (ResNet-101) and input resolution (1024 × 1024 pixels) whenever possible, particularly for the efficiency evaluation. For accuracy comparison, we report the original results as published, which represent each method’s best performance under their respective optimal settings.

4.1.1. DOTA-V1.0

Table 4. Detection accuracy comparison between VODet and mainstream detectors on DOTA-v1.0.

Method	Back-Bone	PL	BD	BR	GTF	SV	LV	SH	TC	BC	ST	SBF	RA	HA	SP	HC	mAP (%)
Two-Stage:
SCRDet [7]	R-101	90.00	80.63	52.07	68.37	68.34	60.33	72.42	90.86	87.92	86.84	65.00	66.69	66.23	68.25	65.22	72.61
Gliding Vertex [23]	R-101	89.66	84.98	52.27	77.32	73.02	73.15	86.80	90.72	79.03	86.79	59.56	70.89	72.95	70.87	57.33	75.02
CSL [13]	R-152	90.26	85.54	54.66	75.33	70.42	73.49	77.63	90.86	86.13	86.67	69.58	68.02	73.81	71.12	68.91	76.17
ReDet [20]	ReR-50	88.81	82.48	60.83	80.82	78.34	86.06	88.31	90.87	88.77	87.03	68.65	66.90	79.26	79.71	74.67	80.10
DGRL [21]	R-101	89.29	85.05	59.70	81.03	79.18	85.76	88.75	90.87	87.30	87.16	70.59	68.68	79.19	79.60	76.18	80.55
Single-Stage:
BBAVectors [25]	R-101	88.63	84.06	52.13	69.56	78.26	80.40	88.06	90.87	87.23	86.39	56.11	65.62	67.10	72.08	63.96	75.36
R3Det [8]	R-152	89.78	84.07	48.09	66.79	78.77	83.25	87.85	90.80	85.39	85.52	65.69	62.66	67.55	78.54	72.63	76.47
PolarDet [41]	R-101	89.63	87.08	48.12	70.96	78.54	80.32	87.46	90.74	85.64	86.85	61.65	70.34	71.90	73.07	67.13	76.64
DCL [14]	R-101	89.27	83.58	53.52	72.78	79.02	82.57	87.32	90.69	86.61	86.99	67.48	66.86	73.31	70.58	70.01	77.37
GWD [16]	R-152	89.08	84.30	55.31	77.55	76.93	70.26	83.93	89.77	84.53	86.08	73.45	67.75	72.62	75.74	74.19	77.43
KLD [17]	R-50	88.91	85.23	53.64	81.23	78.20	76.99	84.58	89.50	86.84	86.38	71.69	68.06	75.95	72.23	75.42	78.32
S2A-Net [12]	R-101	89.30	84.09	56.97	79.19	80.20	82.94	89.23	90.84	84.64	87.63	71.68	68.25	78.56	78.18	65.57	79.15
AO2-DETR [42]	R-50	89.93	84.50	56.88	74.85	80.88	83.45	88.45	90.89	86.14	88.53	63.23	65.11	79.07	82.86	73.48	79.22
SDDLA [43]	R-101	89.51	83.81	59.60	81.92	80.39	84.79	88.60	90.81	85.98	87.50	71.41	69.42	78.44	78.28	68.38	79.92
SARFA-Net [19]	R-50	89.81	85.24	57.08	77.57	81.06	83.08	88.86	90.86	84.90	88.73	72.11	71.11	79.29	78.19	73.53	80.09
R3Det-GWD [16]	R-101	89.64	84.99	59.28	82.15	78.99	84.81	87.68	90.23	86.52	86.87	73.45	67.75	76.94	79.24	74.90	80.23
DFDet [44]	R-101	89.80	84.47	58.68	80.37	82.40	85.18	88.56	90.86	86.65	86.96	70.78	67.32	81.04	77.71	72.19	80.20
VODet (ours)	R-101	90.05	84.82	57.63	79.60	81.46	85.77	89.90	90.78	86.20	88.47	70.52	70.40	77.64	78.34	74.95	80.44

Bold: top performance; underlined: second-best. R-101/50/152: ResNet with different depths; ReR-50: rotation-equivariant ResNet-50.

compares VODet with current mainstream methods on DOTA-v1.0. For the comparison, oriented object detection algorithms are grouped into single-stage methods and two-stage methods. Single-stage detectors immediately output bounding box locations and categories, while two-stage detectors generate region proposals in the first stage, followed by classification and regression in the second.

As shown in the table, VODet obtains the highest accuracy for ships (SH) and the second-highest for planes (PL), small vehicles (SV), and large vehicles (LV). Notably, small vehicles and ships are typically densely arranged targets, while large vehicles belong to the category of large-aspect-ratio targets. This further demonstrates that VODet exhibits excellent detection capabilities for both dense targets and large-aspect-ratio targets. Among all single-stage detectors, VODet achieves the best mAP, surpassed only by the two-stage method DGRL. These results prove that the vertex offset mechanism, through continuous orientation modeling, effectively mitigates the loss of discontinuity caused by angular periodicity and can precisely capture geometric differences in predictions relative to ground truths, highlighting the advantage of our approach for oriented object detection.

The detection performance varies across different categories. For example, SH and TC achieve higher accuracy, 89.90% and 90.78%, respectively, while lower accuracy is observed on BR (57.63%) and SP (78.34%). This disparity is primarily due to class imbalance and inherent visual complexity, a phenomenon also observed in other state-of-the-art methods (e.g., DGRL: 59.70% on BR).

To gain deeper insights, we present the confusion matrix in Figure 12. As can be seen, several confusion patterns emerge between visually similar category pairs. For example, PL and HC are confused due to their similar aircraft shapes. GTF and SBF are confused because both are large field-type objects. SV and LV are confused as they share similar appearances except for scale. SH and HA are confused, possibly due to their spatial proximity in port scenes. In addition, background false positives occur for PL, BR, SV, LV, SH, ST, HA, and SP, indicating that the model sometimes mistakes complex background structures for these categories. False negatives are present across all categories, reflecting the detection difficulty of each class. These results confirm that VODet performs well on most categories, with remaining errors mainly caused by class imbalance, visual similarity, and background interference.

4.1.2. HRSC2016

Table 5. Detection accuracy of various detectors on HRSC2016.

Methods	R3Det [8]	CSL [13]	DAL [45]	S2A-Net [12]	SARFA-Net [19]	KLD [17]	SDDLA [43]	DFDet [44]	VODet (Ours)
Backbone	R-101	R101	R101	R101	R50	R50	R101	R101	R101
AP(%)	89.26	89.62	89.77	90.17	90.40	89.97	90.46	90.38	90.42

Bold indicates the highest AP value.

presents VODet’s detection accuracy relative to other state-of-the-art detectors on HRSC2016. HRSC2016 contains only the ‘ship’ category, in which the objects often exhibit extreme aspect ratios and arbitrary orientations. The results demonstrate that VODet’s detection performance is among the best, achieving an AP of 90.42%, second only to SDDLA’s 90.46%.

Figure 13 illustrates some detection examples of VODet on HRSC2016. Since ships are slender with large aspect ratios, detection effectiveness is easily affected. The proposed VODet can effectively address this challenge, achieving accurate detection.

4.1.3. UCAS-AOD

To further assess VODet’s performance, we performed experiments on the UCAS-AOD dataset, with outcomes summarized in

Table 6. Detection accuracy of various detectors on UCAS-AOD.

Method	Car	Airplane	mAP (%)
RoI Trans. [11]	88.04	90.01	89.03
S2A-Net [12]	89.30	90.16	89.73
DAL [45]	89.25	90.49	89.87
SASM [46]	89.56	90.42	90.00
RIDet-O [47]	88.89	90.33	89.61
SARFA-Net [19]	89.50	90.60	90.05
VODet (ours)	89.67	90.54	90.11

Bold indicates the best performance.

. Table 6 shows that VODet ranks first for cars and second for airplanes, with values of 89.67% and 90.54%, respectively, and achieves the best mAP, demonstrating the effectiveness of the proposed method. Figure 14 presents sample detection results.

4.1.4. Efficiency Comparison

To comprehensively evaluate the trade-off between detection accuracy and computational efficiency, we compare VODet with several state-of-the-art methods on the DOTA-v1.0 dataset in terms of mAP (%), FPS (img/s), FLOPs (G), and parameters (M), as shown in

Table 7. Accuracy and efficiency comparison with other methods on DOTA-v1.0.

Method	FLOPs(G)	Parms(M)	FPS	mAP(%)
Gliding Vertex [23]	269.1	54.2	17.1	75.02
DGRL [21]	265.8	82.7	17.8	80.55
S2A-Net [12]	254.4	47.5	23.0	79.15
DFDet [44]	210.2	48.6	24.5	80.20
VODet (ours)	192.3	51.5	26.7	80.44

. Since it is difficult to obtain the inference speed of all detectors under identical configurations, we select representative one-stage and two-stage detectors that use ResNet-101 as the backbone for a fair comparison. The input image resolution is uniformly set to 1024 × 1024 pixels.

As shown in Table 7, VODet achieves 80.44% mAP at 26.7 FPS, surpassing all compared single-stage methods in both accuracy and speed. It consumes the lowest FLOPs (192.3 G) among all compared methods. Notably, while DGRL achieves slightly higher accuracy, VODet runs substantially faster with 73.5 G fewer FLOPs. All these results indicate that VODet achieves a favorable balance between detection accuracy and computational efficiency.

4.2. Failure Case Analysis

To further understand the limitations of VODet, we present representative failure cases in Figure 15. In the left image, the detection of planes is affected by severe shadow interference, which obscures part of the target region and leads to a missed detection of one plane. In the right image, the presence of heavy haze significantly reduces image quality, resulting in blurred boundaries of a ship target and causing the detector to fail in localizing it. These failures are primarily caused by weakened feature representations under degraded imaging conditions, which hinder the vertex offset regression and direction classification modules. This observation suggests that integrating image restoration techniques [26,27] as a preprocessing step could potentially enhance feature quality and improve detection robustness in such challenging scenarios. We consider this a promising direction for future work.

5. Conclusions

This paper proposes VODet, an oriented object detection framework specifically designed to address the boundary discontinuity and geometric inconsistency challenges in remote sensing images. Unlike traditional angle-based or multi-vertex regression methods, VODet introduces a continuous coupled vertex offset representation that regresses only two offset ratios relative to the horizontal enclosing box, directly reconstructing the four vertices of the rotated bounding box while enforcing geometric consistency. A lightweight direction classifier is further incorporated to handle near-horizontal critical cases. In addition, we design an enhanced backbone with the proposed HSAM and FPN to enrich multi-scale feature representation and adopt a task-specific matching rule with MPDIoU loss to improve localization accuracy. Extensive experiments validate that VODet attains competitive detection performance, with particular strengths in handling large aspect ratios and densely packed objects. Moreover, VODet achieves the highest inference speed and the lowest computational complexity among all compared methods, demonstrating a favorable accuracy-efficiency trade-off. The results confirm that our vertex offset paradigm effectively overcomes the limitations of angular parameterization. Furthermore, while VODet performs well on standard benchmarks, its robustness under degraded conditions such as cloud cover and haze remains to be explored. Integrating image enhancement and restoration techniques to improve detection reliability in such scenarios is a promising direction for future work. In addition, although VODet has been validated on aerial remote sensing datasets, its generalization to other domains (e.g., natural images or street-view scenes) has not been investigated. Exploring cross-domain adaptation for oriented object detection is another important direction for future research.