OShipNet: Occlusion Ship Detection Based on Multidomain Fusion and Multiscale Refinement

Abstract

The growth in international trade has precipitated operational demands on port facilities, mandating the development of advanced intelligent monitoring systems. Existing ship detection algorithms struggle with feature confusion and difficulty in extracting contextual features under occlusion, which reduces the discriminability between object features and background noise. This leads to positional misalignment and mismatching of similar targets, which reduce the detection accuracy. To resolve this, we propose OShipNet, an architecture engineered to optimize feature fusion and refinement for occluded ship detection. First, we design the OShipNeXt backbone network, which provides complementary feature representation in frequency and spatial domains. This approach enables the reconstruction of global–local semantic associations for occluded objects, enhancing feature representation and improving detection accuracy. Secondly, to further refine target boundaries, we develop a Multiscale Pooling Attention Module (MSPAM) to enhance contextual awareness and better capture occluded edge features. Furthermore, we propose a dual-path cooperative loss function that mitigates the effects of low-quality bounding boxes. Comprehensive evaluations on the MVDD13 dataset demonstrate the robustness of OShipNet, which achieved 94.98% mAP@50 and 84.37% mAP@50-95, demonstrating advantages over existing object detection methods and establishing an effective framework for intelligent port monitoring.

1. Introduction

With rapid advances in artificial intelligence, computer vision-based ship detection technology significantly enhances real-time monitoring of ship positions and status, improving dispatch efficiency. Specifically, ship detection technology plays a critical role in intelligent collision avoidance [1] and ship tracking [2]. Currently, researchers are exploring intelligent port ship monitoring using UAVs [3] and remote sensing images [4]. UAVs and remote sensing imagery have inherent limitations (endurance, airspace control, cloud interference, low resolution, and latency); their prohibitive computational latency precludes deployment within expansive, zero-delay terrestrial maritime surveillance frameworks. Therefore, the most widely used approach remains the direct use of shore-based surveillance camera networks for real-time ship monitoring. Existing deep learning-based shore-based monitoring methods like ALF-YOLO [5] and PJ-YOLO [6] have achieved limited progress, as they struggle to address ship occlusion issues.

However, in busy ports and major shipping lanes, the high density of vessel traffic often results in minimal spacing between vessels. This leads to frequent occlusions and imposes greater demands on detection models. Specifically, when an object is partially or fully obscured by other objects, the primary challenge in recognition and localization lies in preserving discriminative feature representations in the presence of occlusion and environmental disturbances. Figure 1 illustrates two common ship occlusion scenarios. Figure 1a shows inter-class occlusion between ships of similar categories. Because marine vessels often exhibit strong visual homogeneity in color, texture, and geometry, partial occlusion can easily lead to feature confusion. This challenges the model’s ability to differentiate individual instances based on limited visible regions. Figure 1b represents occlusion induced by perspective. In maritime environments, a single image often contains ships of varying sizes and categories due to the ocean’s wide field of view. Perspective effects create scale ambiguity, where distant ships (occupying a larger visual scale) and closer, smaller ships exhibit similar visual characteristics. This ambiguity exacerbates detection challenges, as models struggle to resolve spatial relationships and accurately distinguish between ships.

Figure 1. Occlusion between ships under two different conditions: (a) shows occlusion between similar classes, while (b) shows occlusion caused by perspective.

Researchers have conducted research on ship occlusion detection. It has been shown that mosaic data augmentation and the improved YOLO algorithm can effectively improve occlusion detection accuracy in simulated environments [7]. In [8], the combination of global intensity and local texture anomaly maps improves the ability to distinguish ships from surrounding waters, leading to more accurate identification. Wang proposed an enhanced YOLOv5 algorithm to reduce false detection of overlapping ships by featuring receptive field expansion and rapid multiscale feature aggregation [9]. YOLO-Vessel [10] builds on YOLOv7 by incorporating an efficient layer aggregation network for more comprehensive feature fusion.

Although researchers have recorded some achievements in ship detection under complex conditions, there are still many challenges. Specifically, these detection methods typically rely on static convolutional kernels and fixed receptive field designs in the feature processing stage. As a result, they are highly sensitive to feature confusion caused by densely packed ship targets or complex backgrounds, lack dynamic context modeling capabilities, and struggle to distinguish ships with similar textures from background pseudo-targets. They also weaken the explicit representation of important local features, resulting in the loss or distortion of target feature information. Such visual ambiguities severely impair algorithmic precision regarding bounding box regression and semantic classification, ultimately degrading performance. Overall, effective feature extraction and representation are important for achieving high-precision detection of occluded ships.

In view of the above limitations, the objective of this study is to develop an occlusion-aware ship detection framework for complex maritime scenes captured by shore-based surveillance cameras. To this end, we propose OShipNet, a dedicated detection framework designed to enhance feature discrimination and localization robustness for ships under dense occlusion, scale variation, and background interference. Specifically, its backbone, OShipNeXt, consists of four hierarchical OShipStages with OShipBlocks to progressively enlarge the receptive field and strengthen the integration of global and local information through spatial–frequency collaborative representations, thereby improving the distinction of overlapping targets. In addition, the proposed Multiscale Pooling Attention Module (MSPAM) is introduced to enhance feature representation and dynamically recalibrate ship-related responses in occluded scenes by combining multiscale processing with large-kernel convolution attention. Furthermore, a dual-path optimization loss is designed to improve both localization and classification performance, where the point calculation factor and dynamic gradient gain distribution factor enhance bounding box regression, while the Exponential Moving Average Sliding Sample Weighting Loss (EMALoss) improves category prediction. To validate the effectiveness of the proposed framework, experiments are conducted on the MVDD13 dataset, which contains multiple ship categories in realistic maritime scenes, and comparisons are performed against representative ship detection methods. Finally, the contributions of this paper are summarized as follows:

• We propose a new backbone network, OShipNeXt, to alleviate the semantic discontinuity caused by occlusion. By integrating complementary representations from multiple domains, the backbone improves the preservation of visible local structures and strengthens global semantic perception for partially occluded ship targets.
• We introduce a Multiscale Pooling Attention Module (MSPAM) to enhance contextual interaction and scale adaptation in complex occlusion scenarios. In addition, a dual-path optimization loss is developed to improve bounding-box regression and classification stability for heavily overlapping and low-confidence targets.
• Based on the above designs, we establish a unified framework, OShipNet, for occluded ship detection in complex maritime scenes. Extensive experiments on the MVDD13, SeaShip7000, and Crowdhuman datasets demonstrate the effectiveness, robustness, and cross-scenario generalization capability of the proposed framework.

2. Related Work

2.1. Ship Detection

While the emergence of CNNs fundamentally transformed visual recognition [11], contemporary maritime surveillance algorithms heavily depend on standard bounding-box predictors like YOLO [12], DETR [13], and Faster R-CNN [14]. Strategies incorporating GANs [15] or utilizing dilated convolutions [16] successfully elevate detection metrics within typical, unobstructed environments. The integration of global intensity and local texture anomaly maps enhances the discrimination between ships and surrounding waters, thereby improving identification accuracy [8]. A refined YOLOv5 framework, developed by Wang [9], successfully minimizes erroneous bounding boxes among heavily intersecting marine targets. It achieves this robustness through the integration of rapid spatial pyramid pooling and specialized receptive field amplification mechanisms. Ship-YOLO [17] uses a miniature object-enhanced pyramid and a dynamic task-aligned head to achieve accurate ship localization and classification. To effectively handle arbitrarily aligned targets, the RYM architecture [18] extracts built-in orientation cues. Systematically factoring these angular characteristics allows for highly precise multiangle vessel localization. However, the inherent semantic characteristics of ships require an integrated analysis of both local hull structures and contextual features when detecting occluded ships. This joint inference allows for comprehensive ship identification by compensating for the partial information loss caused by occlusion. Traditional ship detection models, which typically lack mechanisms for fusing structural and contextual cues, therefore exhibit limited accuracy in such scenarios.

2.2. Occluded Ship Detection

In real-world scenarios, not all objects are presented in a complete view, and occlusions are very common. While human vision can easily identify occluded objects even under severe occlusion, deep learning detectors experience a significant drop in accuracy due to the presence of occlusion. Therefore, whether the occluded object can be detected has become an important method to verify the robustness of the model.

Conventional detection frameworks maintain high accuracy under minimal obstruction (under 10%), effectively leveraging remaining visible features for successful target identification [19]. However, as the degree of occlusion increases to 50%, the error rate of traditional methods also rises significantly [20]. With the deepening of research, researchers have found that the traditional method, which relies solely on appearance features, has limitations. Occlusion reduces the amount of information that can be used to identify the target, resulting in changes in the shape, contour, and other features of the target, and increasing the difficulty of recognition. Therefore, in the presence of occlusion, contextual information becomes critical. The detector must use environmental cues to infer the presence of the occluded object. This requires the model to have strong semantic understanding and reasoning capabilities. To mitigate mutual blocking and dense clustering among tiny marine vessels, the YOLO-SWD architecture [21] customizes a YOLOv5 baseline, significantly boosting interference resilience. Similarly, the PPGS-YOLO thermal framework [22] incorporates soft non-maximum suppression (Soft-NMS), preserving overlapping candidate boxes to elevate overall localization robustness. With the introduction of the attention mechanism [23], transformer-based detectors have made significant breakthroughs, particularly the Swin Transformer [24]. By adopting the sliding window attention mechanism, it effectively addresses the problem of insufficient information interaction between different parts of the image, thereby improving the model’s ability to capture contextual relationships within the image. Since then, transformer-based occlusion detectors have emerged, such as the Swin Transformer for pedestrian occlusion detection [25,26]. Although frameworks like AodeMar [27] successfully utilize Swin Transformers alongside Coordinate Attention to identify obscured vessels, widespread adoption of self-attention mechanisms remains hindered. Specifically, background noise and visual blockages often distort global attention mappings, subsequently deteriorating fine-grained feature extraction. Furthermore, their inherent quadratic processing overheads severely limit real-world edge-device implementation.

3. Proposed Method

3.1. Overall Architecture

Occluded ship detection can be viewed as learning a mapping from an input image to object categories and bounding boxes under incomplete visual observations. Let I denote an input image, and let $\widehat{Y}={\left\{({\widehat{b}}_i,{\widehat{c}}_i)\right\}}_{i=1}^N$ represent the predicted bounding boxes and categories. The overall detection process of OShipNet can be conceptually formulated as:

$$\widehat{Y}=H\left(F_{ctx}\left(F_{rep}\left(I\right)\right)\right)$$

(1)

where $F_{rep}(·)$ denotes the feature representation stage, $F_{ctx}(·)$ denotes the contextual refinement stage, and $H(·)$ denotes the detection head.

Compared with conventional detector designs that mainly improve a single stage, OShipNet is developed following an occlusion-oriented coordinated design principle. Specifically, the proposed framework is organized around three coupled difficulties in occluded ship detection, namely incomplete local structural cues, contextual and scale ambiguity, and unstable optimization caused by heavily overlapping targets. Accordingly, the proposed framework can be summarized as:

$$D_{occ}=\left\{F_{rep}^{occ},F_{ctx}^{occ},L^{occ}\right\}$$

(2)

where $F_{rep}^{occ}$ aims to preserve discriminative visible structures under partial occlusion, $F_{ctx}^{occ}$ aims to recover contextual and multiscale semantic cues around occluded instances, and $L^{occ}$ aims to provide stable supervision for ambiguous and overlapping samples during training.

As illustrated in Figure 2, OShipNet is built upon the one-stage object detection framework YOLOv11 and further tailored for occluded ship detection in complex maritime environments. The framework consists of three tightly coupled parts: an occlusion-aware backbone named OShipNeXt, a multiscale contextual refinement module named MSPAM in the neck, and a dual-path optimization loss for localization and classification. Specifically, OShipNeXt is responsible for strengthening feature representation under incomplete visibility, MSPAM enhances contextual interaction and scale adaptation, and the dual-path loss improves optimization stability when targets are heavily overlapped. Through this coordinated design, OShipNet improves its ability to capture discriminative cues under severe occlusion, perspective interference, and complex background interference.

Figure 2. Overall Architecture: (a) represents the overall network architecture; (b–f) represent the specific modules in the network architecture.

3.2. OShipNeXt

In accordance with the works of [28,29], a novel backbone network, OShipNeXt, was conceptualized. This backbone network employs a multidomain joint approach, fusing locational and spectral characteristics to construct comprehensive feature representations. As shown in Figure 2a, OShipNeXt is divided into 1 CBS (Figure 2d) and 4 OShipStages (Figure 2c). Each OShipStage consists of 1 downsampling stage and 2 OShipBlocks. From a conceptual perspective, the proposed OShipNeXt can be expressed as:

$$X_{rep}=MLP\left(Norm\left(GELU\left(Concat(f_1\left(X\right),f_2\left(X\right),\dots ,f_6\left(X\right)\right))+X\right)\right)+X$$

(3)

where X denotes the input feature, $f_i(·)$ denotes the transformation of the i-th branch, and $X_{rep}$ denotes the refined representation. The branch functions are designed to capture complementary cues from different perspectives, including identity-preserved information, local spatial patterns, enlarged receptive-field responses, frequency-aware structures, and receptive-field attention. In this way, OShipNeXt enhances semantic continuity for partially visible targets while retaining fine-grained local details that remain observable under occlusion.

For each feature entering the OShipStage, the feature representation is given by $[B,\frac{C}{2},2H,2W]$, where $[B,C,H,W]$ denote the batch size, channels, the height and width of the feature. As illustrated in Figure 2c, the features will first go into the downsampling stage and the feature size $[B,C,H,W]$ is obtained by 2× downsampling. Specifically, to prevent information degradation typical in standard ConvNeXt [30] spatial reductions, a normalization layer and the GELU activation function are introduced at our downsampling stage. The normalization layer ensures the stability of the feature distribution, mitigating problems such as vanishing or exploding gradients and improving the training stability of the network. The GELU activation function improves the network’s ability to adapt to occluded environments through its smooth nonlinear mapping. This improved downsampling stage helps the model better retain essential features and reduces the loss of information for occluded objects.

Next, after downsampling stage, the feature with size $[B,C,H,W]$ is fed into the OShipBlock for refined feature processing. To facilitate parallel multidomain processing, the incoming tensor within the OShipBlock undergoes channel-wise separation. This systematic splitting yields six independent subsets, each characterized by the geometric shape $[B,\frac{C}{8},H,W]$.

The channel split is introduced to allocate the representation capacity of the block to multiple complementary subspaces with different receptive characteristics. Specifically, the split branches are responsible for preserving original information, modeling local spatial textures, enlarging directional receptive fields, capturing frequency-aware structural cues, and performing adaptive receptive-field emphasis. Such a design improves branch diversity while retaining sufficient channel width for each branch. In this way, the block maintains a balance between representation specialization and feature stability, which is important for partially occluded ship targets whose discriminative cues are often fragmented across local structure, contour context, and low-frequency semantics.

More specifically, for the first part of the features, no operations are applied, allowing it to retain the original information. In the second part, a depth-wise separable convolutional operation of $3\times 3$ size is adopted on input feature maps, with independent convolution executed for each channel. This approach reduces the computational complexity compared to standard convolution while preserving spatial information [31]. Inspired by GoogLeNet [32], the $11\times 11$ large-kernel convolution is decomposed into $1\times 11$ and $11\times 1$ convolution kernels. The third and fourth parts of the features undergo these convolutions separately. This approach allows the model to achieve a larger receptive field while reducing the computational cost associated with large-kernel convolutions.

The fifth and sixth feature parts are processed by $3\times 3$ wavelet convolution [33] after aggregation. We decompose the input feature tensor into four channel-wise frequency elements with halved spatial resolution, process each element via small-kernel depth-wise separable convolution, reconstruct features through inverse discrete wavelet transform, and introduce residual connections to retain input information. This design supports fine-grained multiscale feature extraction and efficient low-frequency feature capture.

For the seventh and eighth features, we aggregate them and apply Receptive-Field Attention Convolution (RFAConv) [34] to introduce spatial attention in the model. By taking into account the importance of features within the receptive field, a dynamic weight adjustment mechanism is used to detect target features at different positions, even under occlusion conditions. The unfold method, which extracts spatial features from the receptive field, further enhances attention to these features, improving network performance. The specific convolutional computation is as follows:

$$\begin{array}{c}\hfill A_{rf}=Softmax\left(Con{v}_{group}^{1\times 1}\left(AvePool\left(X_{input}\right)\right)\right)\end{array}$$

(4)

$$\begin{array}{c}\hfill R_{rf}=ReLU\left(Norm\left(Con{v}_{group}^{5\times 5}\left(X_{input}\right)\right)\right)\end{array}$$

(5)

$$\begin{array}{c}\hfill X_{RFA\_out}=A_{rf}\times R_{rf}\end{array}$$

(6)

$A_{rf}$ represents the attention feature map and $R_{rf}$ represents the receptive field spatial feature map. By highlighting the distinct contributions of different elements in the receiving field and giving greater emphasis to spatial features, RFAConv improves the network’s critical information capture capability.

We aggregate eight processed feature tensors via channel-wise concatenation to reconstruct a feature tensor with dimensionality $[B,C,H,W]$, which is then passed through a GELU activation layer and fused with the original input via residual skip connection. This fusion strategy retains shallow features for fine-grained detail capture in complex scenarios, and simultaneously enhances the model’s feature representation capability and training efficiency. Finally, the feature is passed through the Norm (Normalization Layer) and the MLP (Multilayer Perceptron) to obtain the final OShipBlock output.

3.3. Multiscale Pooling Attention Module (MSPAM)

In ship detection scenarios with frequent occlusion, the correlation between local edge features and global structural information is often weak, making it difficult to infer object presence and boundaries from partially visible observations. In addition, single-scale feature representations extracted by the backbone network lack robustness to scale variations caused by occlusion and perspective changes. To address these limitations, we propose a Multiscale Pooling Attention Module (MSPAM), as shown in Figure 3, which enhances feature representation by jointly leveraging hierarchical context aggregation and adaptive attention refinement. Given an input feature map X, we first obtain a channel-adjusted feature $X^{\prime }$ via a $1\times 1$ convolution. Then, multiple pooling operations with different receptive fields are applied to capture contextual information at different scales:

$$P_k\left(X^{\prime }\right)=MaxPoo{l}_k\left(X^{\prime }\right),k\in \{1,2,3\}$$

(7)

The original feature and pooled features are concatenated along the channel dimension to form a multiscale aggregated representation:

$$X_{agg}=Concat\left(P_0\left(X^{\prime }\right),P_1\left(X^{\prime }\right),P_2\left(X^{\prime }\right),P_3\left(X^{\prime }\right)\right),P_0\left(X^{\prime }\right)=X^{\prime }$$

(8)

which effectively enlarges the contextual support region of each spatial location and provides complementary information for occluded regions. Based on this, the overall architecture of MSPAM can be summarized as:

$$X_{ctx}=A_{ms}\left(X_{agg}\right)$$

(9)

where $A_{ms}(·)$ denotes the subsequent multiscale convolution-attention refinement.

Figure 3. The architecture of multiscale pooling attention module (MSPAM).

Figure 3. The architecture of multiscale pooling attention module (MSPAM).

Specifically, the aggregated feature $X_{agg}$ is further processed using a series of depthwise separable convolutions with asymmetric kernels ($1\times 3$, $3\times 1$) and expanded kernels ($1\times 5$, $5\times 1$), which approximate larger receptive fields while maintaining computational efficiency. In addition, dilated convolutions are introduced to further enlarge the effective receptive field, enabling the model to capture richer global contextual dependencies. After convolutional transformation, the feature is mapped to generate attention weights.

Through this design, MSPAM simultaneously achieves two objectives: alleviating the loss of discriminative information caused by missing information under occlusion, and adaptively recalibrating multiscale feature responses to improve robustness to scale variation and dense spatial overlap. As a result, the proposed module enables the network to better capture both local visible details and global structural cues, thereby improving feature discriminability and detection confidence in complex occlusion scenarios. This design can be further interpreted from a feature propagation perspective: the multiscale pooling paths introduce parallel context flows with different receptive spans, which reduces the dependency on a single-scale feature pathway and improves the continuity of feature propagation under partial occlusion.

3.4. Dual-Path Optimization Loss Function

From an optimization perspective, occluded object detection can be characterized by two major challenges: (1) ambiguous geometric correspondence caused by overlapping bounding boxes, and (2) imbalanced gradient contributions induced by samples of varying difficulty. These issues lead to unstable gradient signals and suboptimal convergence when conventional loss functions are applied.

To address this, the proposed dual-path loss is designed to explicitly regulate both geometric alignment and gradient distribution. Specifically, the localization term focuses on improving the geometric identifiability under overlap, while the classification term dynamically adjusts the gradient contribution across samples of different confidence levels, thereby stabilizing the overall optimization process. In this work, the optimization objective of OShipNet is formulated as:

$$L_{det}^{occ}=L_{loc}^{OIoU}+L_{cls}^{EMA}$$

(10)

where $L_{loc}^{OIoU}$ denotes the proposed overlap-aware localization loss and $L_{cls}^{EMA}$ denotes the dynamically weighted classification loss. The localization term is developed to improve bounding-box regression when target geometry becomes ambiguous under heavy overlap, while the classification term enhances supervision for hard and uncertain samples during training.

For bounding-box regression, the baseline model uses CIoU as the regression loss function, which simultaneously considers the center points of bounding boxes and aspect ratio difference between predicted boxes and ground-truth annotations [35]. With the aspect ratio penalty term, CIoU provides effective optimization of both shape consistency and positional alignment. Nevertheless, in occlusion-heavy scenarios, CIoU cannot adequately model the complex geometric relations among partially overlapped targets, which reduces its discriminative capability for highly overlapping bounding boxes. In addition, when the center points of multiple occluded targets are very close, the optimization effect of the center-distance term becomes weakened, thereby limiting localization precision.

To address this issue, we propose a new bounding-box regression loss, termed Occlusion Intersection over Union (OIoU). OIoU consists of three parts, namely the IoU term, a point calculation factor, and a dynamic gradient gain distribution factor, as defined in Equation (11).

$$\begin{array}{c}\hfill L_{OIoU}=1-IoU+F_{point}+F_{dynamic},IoU={\displaystyle \frac{A\cap B}{A\cup B}}\end{array}$$

(11)

First, considering the slow convergence speed of the existing IoU loss algorithm, we use the point calculation factor for optimization, as shown in Equation (12), $W_{img}$ and $H_{img}$ represent the width and height of the image feature, and the equations of $d_1^2$ and $d_2^2$ are shown as Equations (13) and (14):

$$\begin{array}{c}\hfill F_{point}={\displaystyle \frac{d_1^2+d_2^2}{W_{img}^2+H_{img}^2}}\end{array}$$

(12)

$$\begin{array}{c}\hfill d_1^2={(x_1^{pre}-x_1^{gt})}^2+{(y_1^{pre}-y_1^{gt})}^2\end{array}$$

(13)

$$\begin{array}{c}\hfill d_2^2={(x_2^{pre}-x_2^{gt})}^2+{(y_2^{pre}-y_2^{gt})}^2\end{array}$$

(14)

As shown in Figure 4, $(x_1^{pre},y_1^{pre})$, $(x_2^{pre},y_2^{pre})$ represent the upper-left and lower-right coordinates of the prediction bounding box, while $(x_1^{gt},y_1^{gt})$, $(x_2^{gt},y_2^{gt})$ represent those of the ground truth bounding box. The point calculation factor can be interpreted as an explicit coordinate-level constraint that supplements the region-based IoU metric. While IoU provides a coarse measure of overlap, it suffers from gradient sparsity when the predicted and ground-truth boxes have limited intersection. In contrast, the point distance term introduces a dense and continuous gradient signal in the coordinate space, ensuring stable optimization even in low-overlap regimes.

Figure 4. Point calculation factor diagram.

Secondly, to address the ambiguous optimization direction caused by overlapping center points between targets, we introduce a dynamic gradient gain distribution factor. As shown in Equation (15), by using an adaptive penalty coefficient and a nonlinear mapping function, this mechanism balances the gradient contributions of high- and low-quality samples to discriminate occluded targets.

$$\begin{array}{c}\hfill F_{dynamic}={\displaystyle \frac{\beta }{\delta {\alpha }^{\beta -\delta }}}·(1-IoU)·e^{\frac{{\rho }^2}{C_w^2+C_h^2}}\end{array}$$

(15)

$$\begin{array}{c}\hfill \rho ={\displaystyle \frac{{(x_2-x_1)}^2+{(y_2-y_1)}^2}{4}}\end{array}$$

(16)

To be specific, $\beta$ represents the rate of IoU variation, balancing the gradient contributions of high-quality samples and low-quality samples. $\alpha$ controls the fundamental shape of the weight curve, determining the sensitivity of the weight to IoU changes. $\delta$ dynamically scales the weight, adaptively adjusting the loss magnitude to prevent numerical instability. Here, $C_w$ and $C_h$ denote the width and height of the minimum enclosing bounding box, which eliminates convergence-impeding gradient anomalies of the original IoU metric. $\rho$ denotes the squared Euclidean distance between the center point of the predicted bounding box and that of the ground-truth bounding box. The dynamic factor can be viewed as a gradient reweighting mechanism that redistributes optimization emphasis based on the quality of predictions. Specifically, by introducing a nonlinear modulation term dependent on IoU and spatial distance, the proposed formulation amplifies gradients for ambiguous or poorly localized samples while suppressing those for already well-aligned predictions. This mechanism effectively mitigates the gradient domination problem, where easy samples contribute disproportionately to the overall optimization, and ensures that hard occluded instances receive sufficient training signals.

For classification loss, the commonly used binary cross-entropy (BCE) loss measures the discrepancy between the model’s predicted probability and the true label. The model is trained by minimizing this difference, optimizing its classification performance. The formula is given in Equation (17).

$$\begin{array}{c}\hfill L_{BCE}=-[y\ast \log \left(p\right)+(1-y)\ast \log (1-p)]\end{array}$$

(17)

However, the similarity between ships can significantly impact classification accuracy under occlusion. Traditional BCE loss employs a static weight allocation mechanism, treating the prediction error of all samples equally and lacking dynamic attention to hard samples [36,37]. As a result, the model cannot effectively capture fine-grained feature differences between similar targets in dense occlusion scenarios, resulting in a significant decline in classification robustness. In this paper, a new Exponential Moving Average Sliding Sample Weighting Loss Function (EMALoss) is used to distinguish similar targets effectively.

First, based on the object’s IoU values, the image is divided into three regions: low confidence, transition, and high confidence, each of which is assigned a different modulation weight to improve classification adaptability. Next, different factors are applied to different regions. In the low-confidence region, no additional processing is performed. In the transition region, an exponential amplification factor improves the model’s attention to ambiguous boundary targets. In the high-confidence region, an exponential attenuation factor is introduced to mitigate the risk of overfitting. Meanwhile, the mean IoU is updated in real time using the Exponential Moving Average (EMA) algorithm, smoothing training fluctuations and minimizing the impact of local noise on the loss calculation. Finally, the loss function dynamically adjusts the gradient contribution of samples of varying difficulty by incorporating automatically adjusted modulation weights. The specific calculation method is shown in Equations (18) and (20):

$$M=\left\{\begin{array}{cc}1,\hfill & \mathrm{score}\le {\mathrm{IoU}}_{\mathrm{m}}-0.1\hfill \\ e^{1-{\mathrm{IoU}}_{\mathrm{m}}},\hfill & {\mathrm{IoU}}_{\mathrm{m}}-0.1<\mathrm{score}<{\mathrm{IoU}}_{\mathrm{m}}\hfill \\ e^{-(\mathrm{score}-1)},\hfill & \mathrm{score}\ge {\mathrm{IoU}}_{\mathrm{m}}\hfill \end{array}\right.$$

(18)

$${\mathrm{IoU}}_{\mathrm{m}}^{\left(t\right)}=d·{\mathrm{IoU}}_{\mathrm{m}}^{(t-1)}+(1-d)·\mathrm{IoU}$$

(19)

$$\begin{array}{c}\hfill L_{EMALoss}=Mean(L_{BCE}\ast M)\end{array}$$

(20)

The score denotes the matching value between each predicted bounding box and its paired ground truth annotation, while ${\mathrm{IoU}}_{\mathrm{m}}$ represents the exponentially smoothed average IoU, dynamically estimated during training, as defined in Equation (19), d is the exponential smoothing coefficient.

By automatically adjusting the weights, the proposed classification loss dynamically regulates the gradient contributions of samples with different difficulty levels, which can be interpreted as a piecewise gradient scheduling strategy. Specifically, samples are partitioned into low-confidence, transitional, and high-confidence regions according to their IoU scores, and each region is assigned a distinct gradient scaling behavior. When the IoU falls below the threshold, the corresponding samples are treated as hard examples and their weights are increased, thereby reinforcing the learning of ambiguous and poorly localized instances. In contrast, samples in the low-confidence region maintain baseline gradients to preserve training stability, while those in the transitional region receive amplified gradients to enhance discriminability. For high-confidence samples, a progressive weight decay mechanism is applied to attenuate their gradient contributions, preventing them from dominating the optimization process and reducing the risk of convergence to suboptimal local minima with poor generalization performance.

3.5. Datasets

To evaluate OShipNet, we use three public datasets: MVDD13, Seaship7000, and Crowdhuman, for primary evaluation, cross-dataset transfer validation, and dense-occlusion generalization analysis, respectively. Detailed dataset statistics are provided in Appendix A.

MVDD13: MVDD13 is used as the primary benchmark because it explicitly contains occlusion information and is suitable for ship detection in complex maritime scenes. It includes 25,541 training images, 2837 validation images, and 7095 test images, and is used for training, validation, and comparative evaluation in this work.

Seaship7000: Seaship7000 is used to assess the cross-dataset generalization capability of the proposed model. For this purpose, the dataset is re-split into a fine-tuning set and a validation set at a 1:1 ratio, and comparisons between pretrained and non-pretrained settings are conducted.

Crowdhuman: Crowdhuman [38] is further introduced to evaluate the robustness of OShipNet in highly crowded and heavily occluded scenes beyond the maritime domain. It contains 19,370 images, including 15,000 training images and 4370 validation images, and provides a challenging benchmark due to its dense object layout and severe inter-instance occlusion.

3.6. Evaluation Metrics

In this work, the evaluation metrics include Precision (P), Recall (R), mean Average Precision (mAP), model parameters (Params), floating-point operations (FLOPs), and inference time. Following standard practice in object detection, we report both mAP@50 and mAP@50-95. Specifically, mAP@50 is evaluated at an IoU threshold of 0.5, whereas mAP@50-95 is calculated by averaging the mAP values over IoU thresholds from 0.50 to 0.95 with a step size of 0.05. In addition, Params, FLOPs, and inference time are reported to assess model complexity and computational efficiency. Detailed definitions of these metrics are provided in Appendix B.

4. Experiments

4.1. Experimental Settings

The full set of ablation and comparative experiments was carried out on an Ubuntu 20.04 LTS system equipped with an NVIDIA RTX 3090 24 GB GPU, based on the PyTorch 2.2.0 framework and CUDA 11.8. Table 1 summarizes the hyperparameter configurations used in our experiments.

Table 1. Hyperparameter Configuration.

Hyperparameter	Value
Image Size	$640\times 640$
Epoch	300
Batch Size	32
Learning Rate	0.01
Momentum	0.937
Weight Decay	0.0005
Optimizer	SGD

To ensure a fair comparison, all baseline methods and the proposed model were trained and evaluated under the same experimental conditions, including consistent input resolution, training epochs, batch size, optimizer settings, and data augmentation strategies.

4.2. Comparative Experiments

In order to provide a comprehensive demonstration of the efficacy of the proposed method, a comparative analysis is conducted between OShipNet and various advanced detection models. The comparative set includes two-stage detector Faster R-CNN [14], SSD [39], as well as YOLOv5, YOLOv8, YOLOv11 [40], YOLOv12 [41], and the latest Transformer-based detectors, Swin Transformer [24] and RT-DETR [42]. We conduct a comprehensive evaluation of the proposed OShipNet framework across detection accuracy, computational efficiency and robustness, with results summarized in Table 2. Comparative results show that YOLO-series models achieve a better accuracy–efficiency trade-off than the two-stage Faster R-CNN framework. Our OShipNet achieves state-of-the-art performance on all core metrics via occlusion-aware innovations: it reaches 94.98% mAP@50 and 84.37% mAP@50-95, outperforming YOLOv11 by 1.26% and 2.08%, and yields 1.35%, 2.30%, 1.46%, 2.47% gains on Precision, Recall, mAP@50, mAP@50-95 over the YOLOv12 baseline, especially for occluded targets. Meanwhile, OShipNet outperforms Swin Transformer and RT-DETR by a significant margin on all core metrics, demonstrating the effectiveness of our proposed method. In terms of inference time, OShipNet runs at 18.28 ms per image. Although it is slightly slower than several lightweight YOLO baselines, it remains markedly more efficient than Faster R-CNN and Transformer-based detectors, demonstrating a reasonable trade-off between accuracy and inference efficiency. These results indicate that the proposed method does not seek a universally large margin on all samples, but instead provides consistent improvements under the target problem considered in this work, namely occluded ship detection in complex maritime scenes. Notably, the gain is more evident on the stricter metric mAP@50-95, suggesting that the main benefit of OShipNet lies in more reliable localization and box refinement under challenging conditions.

Table 2. Detection performance comparison between OShipNet and other methods.

Model	Based	Precision	Recall	mAP@50	mAP@50-95	FLOPs	Params	Inference Time
Faster R-CNN [14]	CNN	72.41	75.85	91.18	72.89	124.7 G	41.41 M	36.61 ms
SSD [39]	CNN	85.04	80.09	92.07	75.36	62.1 G	25.35 M	27.19 ms
YOLOv5	CNN	94.57	86.51	93.32	81.13	18.8 G	7.81 M	11.81 ms
YOLOv8	CNN	93.96	87.69	93.70	82.18	23.4 G	9.84 M	14.96 ms
YOLOv11 [40]	CNN	94.27	87.80	93.72	82.29	21.7 G	9.46 M	12.84 ms
YOLOv12 [41]	CNN	94.54	87.26	93.52	81.90	21.2 G	9.23 M	14.01 ms
Swin Transformer [24]	Transformer	71.26	75.20	91.51	74.86	169.9 G	47.45 M	45.57 ms
RT-DETR [42]	Transformer	95.78	88.42	93.31	82.02	186.3 G	60.93 M	41.24 ms
OShipNet (ours)	CNN	95.89	89.56	94.98	84.37	29.6 G	14.17 M	18.28 ms

In addition, we present AP50 metrics for each specific category. As shown in Table 3, OShipNet achieves the best results in most categories, with the highest performance in detecting cargo ships under occlusion. This indicates that OShipNet excels at detecting cargo ships despite occlusion, demonstrating its strong ability to identify objects in various occlusion scenarios. For bulkers and warships, OShipNet did not achieve the highest AP50, but still achieved second-best results with performance close to the best. Bulk carriers exhibit a high degree of visual similarity to cargo ships and oil tankers in terms of shape, texture, and overall structure, which increases inter-class confusion. As shown in the confusion matrix, most misclassified bulk carriers were incorrectly identified as cargo ships or oil tankers. Severe occlusion further reduces the visibility of discriminative cues, thereby making accurate classification more challenging. In addition, compared with some other ship categories, the distinctive characteristics of bulk carriers are relatively subtle, rendering this category more susceptible to feature degradation in complex environments. In summary, the excellent performance of OShipNet provides a new technical path for dense occlusion target detection.

Table 3. Comparison of per-class detection accuracy between OShipNet and other methods.

Categories	Faster R-CNN	SSD	YOLOv5	YOLOv8	YOLOv11	YOLOv12	Swin Transformer	RT-DETR	OShipNet (Ours)
Cargo Ship	96.13	96.51	97.64	97.90	97.84	97.78	96.04	96.38	98.03
Tanker	96.89	97.16	97.98	98.07	98.12	98.00	95.87	97.33	98.52
Bulker	97.08	96.25	97.71	98.04	97.75	97.56	96.93	95.08	97.98
Containership	86.30	86.02	90.77	90.81	90.59	90.91	87.44	92.15	92.94
Tug	86.96	88.39	89.04	89.09	89.13	89.18	86.73	89.52	91.39
Fishing Boat	81.40	82.17	83.31	84.06	84.14	85.03	80.80	84.61	86.02
Drill Ship	94.49	95.79	97.91	97.99	98.01	97.56	96.12	97.72	98.42
Passenger Ship	93.07	93.22	93.68	94.58	94.48	94.14	93.55	93.88	95.20
Cruise Ship	97.63	97.75	98.64	98.48	98.46	98.74	97.61	97.62	98.92
Warship	83.18	86.44	85.24	85.37	86.02	86.52	83.89	88.31	86.82
Sailing Boat	92.78	90.27	93.79	94.45	93.78	94.18	92.38	93.54	95.28
Submarine	87.58	93.33	93.15	94.84	95.18	93.46	89.70	94.27	98.28
Firefighting Ship	91.91	93.59	94.32	94.42	94.81	92.69	92.60	92.60	96.93

To further demonstrate the effectiveness of OShipNet, we present the confusion matrix comparing OShipNet and YOLOv12 on the testing set, as shown in Figure 5. The confusion matrix shows that OShipNet achieves high accuracy in classifying many categories, such as cargo (1404), tanker (895), bulker (1006), and cruise (1065). The large diagonal values indicate that OShipNet effectively identifies these target categories with strong classification performance. In addition, there is less confusion between categories such as cargo and tanker, indicating that the model is more stable in distinguishing between these more similar categories. For categories like warship (72), submarine (61), and firefighting (247), despite the smaller sample sizes, OShipNet still maintains a significant number of correct predictions along the diagonal. This indicates that OShipNet demonstrates a notable ability to recognize smaller categories as well. Finally, we also provide the OShipNet and YOLOv12 Precision-Recall curves, as shown in Figure 6.

Figure 5. Confusion Matrix.

Figure 6. Precision–Recall Curve.

4.3. Visualization

In addition, we conducted visualization experiments on the MVDD13 dataset using the aforementioned mainstream detection algorithms to further evaluate the performance of OShipNet in ship detection. The visualization results are presented in Figure 7. From the visualization results, OShipNet demonstrates clear advantages in several aspects. Firstly, in terms of target localization, its detection bounding boxes exhibit a higher degree of overlap with the ground truth labels, and the bounding boxes are more precise, indicating a stronger target regression capability. In the first set of images, compared to the YOLO series and Transformer-based methods, OShipNet more accurately outlines the ship’s contours and effectively identifies the target ship even in occluded scenarios. Secondly, OShipNet maintains strong detection performance even under perspective occlusion. For instance, in the second set of images, when faced with illumination changes and perspective interference, Faster R-CNN, SSD, and other methods exhibited missed detections or bounding box deviations. In contrast, OShipNet accurately detected the object. Finally, in the third set of images, when dealing with a densely packed ship scenario, OShipNet exhibited relatively good robustness. Compared to other methods such as YOLOv5 and YOLOv8, its detection boxes were more complete, and its target classification was more accurate.

Figure 7. Visualization Results.

To provide a more intuitive interpretation of the proposed method, we further compare the feature responses of YOLOv11 and OShipNet at the Backbone, Neck, and Detection Head stages using heatmap visualization. As shown in Figure 8, the baseline YOLOv11 exhibits relatively scattered activations, and some responses are distributed over background regions rather than being concentrated on the occluded ship targets. By contrast, OShipNet produces more compact and target-consistent responses throughout the network.

Figure 8. GradCAM Visualization Results.

At the Backbone stage, OShipNet shows stronger activation on the main ship body and key structural parts, indicating that the proposed feature extractor is more capable of preserving informative local cues under occlusion. At the Neck stage, the highlighted regions become more continuous and semantically coherent, suggesting that the feature fusion strategy improves the aggregation of multiscale contextual information and suppresses irrelevant background interference. At the Detection Head stage, OShipNet exhibits more target-focused responses with less activation dispersion, which reflects a more stable localization tendency in complex maritime scenes.

From a practical perspective, the advantage of OShipNet is more apparent in difficult visual conditions, including partial occlusion, perspective-induced overlap, and densely distributed ship instances. In these scenarios, the proposed model produces more complete detections, fewer missed targets, and bounding boxes that are more consistent with the ground truth. The Grad-CAM visualizations also show that OShipNet yields more compact and target-focused responses than YOLOv11 across the backbone, neck, and detection head, indicating that the proposed design is better able to preserve discriminative information and suppress irrelevant background responses under occlusion.

4.4. Ablation Study

In this section, we verify the contribution and effectiveness of each module in OShipNet through rigorous ablation experiments. We adopt a multidimensional evaluation system including Precision, Recall, mAP@50, mAP@50-95, FLOPs and Parameters, and keep all other configurations strictly consistent during the experiments to accurately quantify the contribution of each module.

4.4.1. Effect of Modules

To verify the effectiveness of the proposed modules, we conducted a systematic ablation study on the OShipNet framework, and the detailed results are reported in Table 4. Each row in the table corresponds to one ablation configuration based on the baseline model, and the checkmarks indicate which modules are enabled, including the OShipNeXt backbone, the multiscale pooling attention module (MSPAM), and the dual-path loss function. Compared with the baseline model, replacing the original backbone with OShipNeXt improves the mAP@50-95 from 81.90% to 83.62%, demonstrating that the proposed backbone is more effective in extracting fine-grained and discriminative features. When MSPAM is introduced individually, the mAP@50-95 increases to 83.85%, indicating that multiscale pooling and attention operations enhance feature representation capability. Adding the dual-path loss alone improves the mAP@50 from 93.52% to 94.05%, showing its effectiveness in optimizing localization quality. When these modules are further combined, the overall performance is consistently improved. Finally, the full model achieves the best detection accuracy, with Precision, Recall, mAP@50, and mAP@50-95 reaching 95.89%, 89.56%, 94.98%, and 84.37%, respectively, which confirms the complementary benefits of the proposed backbone, attention module, and loss design.

Table 4. Ablation study of the proposed modules.

Baseline	OShipNeXt	MSPAM	Dual-Path Loss	Precision	Recall	mAP@50	mAP@50-95	FLOPs	Params
✓				94.54	87.26	93.52	81.90	21.2 G	9.23 M
✓	✓			94.37	89.08	94.39	83.62	27.8 G	12.10 M
✓		✓		94.77	88.30	94.38	83.85	23.6 G	11.96 M
✓			✓	94.80	88.23	94.05	83.06	21.3 G	9.24 M
✓	✓	✓		94.89	88.18	94.30	84.13	29.5 G	14.16 M
✓		✓	✓	94.50	89.08	94.64	84.34	23.6 G	11.96 M
✓	✓		✓	94.46	89.29	94.64	83.90	27.8 G	12.10 M
✓	✓	✓	✓	95.89	89.56	94.98	84.37	29.6 G	14.17 M

4.4.2. Effect of Different Backbone Networks

To verify the effectiveness of our proposed OShipNeXt backbone, we conducted controlled comparative experiments with different mainstream backbones while keeping all other modules fixed, with results shown in Table 5. OShipNeXt achieves the best performance across all metrics: it outperforms the second-best InceptionNeXt by 0.13% in mAP@50 and 0.21% in mAP@50-95, and delivers 1.75% mAP@50 and 2.97% mAP@50-95 gains over ConvNeXt v1, demonstrating its superior fine-grained localization capability under high IoU thresholds. This performance gain is attributed to the spatial–frequency domain information fusion in OShipNeXt, which enhances the model’s overall feature representation capability. Meanwhile, OShipNeXt shows superior lightweight efficiency in parameter count and computational cost compared with the other three backbone networks. Its resource requirements are significantly lower than those of ConvNeXt v1 (46.9 G FLOPs, 22.55 M Parameters) and InceptionNeXt (38.2 G FLOPs, 18.51 M Parameters).

Table 5. Ablation experiments between different backbone networks.

Backbone	Precision	Recall	mAP@50	mAP@50-95	FLOPs	Params
ConvNeXt v1 [30]	93.26	87.70	93.23	81.40	46.9 G	22.55 M
ConvNeXt v2 [28]	95.26	88.63	94.52	83.79	33.0 G	15.59 M
InceptionNeXt [29]	95.18	88.89	94.85	84.16	38.2 G	18.51 M
OShipNeXt (ours)	95.89	89.56	94.98	84.37	29.6 G	14.17 M

4.4.3. Effect of Proposed Loss Function

In object detection, the overall optimization objective is generally composed of a classification loss and a localization loss, which can be written in a simplified form as Equation (21).

$$L_{det}=L_{cls}+L_{loc}$$

(21)

where $L_{cls}$ denotes the classification loss used to supervise category prediction, and $L_{loc}$ denotes the localization loss used to optimize bounding box regression. Therefore, to comprehensively evaluate the effectiveness of the proposed loss design, we conduct ablation experiments from both aspects. Specifically, for the localization loss, we verify the effectiveness of the proposed point calculation factor and dynamic gradient gain distribution factor through different hyperparameter settings. For the classification loss, we compare different classification loss combinations together with different IoU losses.

To evaluate the effectiveness of the proposed point calculation factor and dynamic gradient gain distribution factor in bounding box regression, we introduce the hyperparameters $\lambda$ and $\mu$ into the localization loss, as defined in Equation (22).

$$L_{loc}=1-IoU+\lambda F_{point}+\mu F_{dynamic}$$

(22)

where $F_{point}$ denotes the point calculation factor and $F_{dynamic}$ denotes the dynamic gradient gain distribution factor. The corresponding ablation results are reported in Table 6. Each row in the table corresponds to one specific hyperparameter setting $(\lambda ,\mu )$.

Table 6. Ablation results of different hyperparameter settings in the proposed localization loss. Each row corresponds to one $(\lambda ,\mu )$ configuration.

Hyperparameter		Precision	Recall	mAP@50	mAP@50-95
λ	μ
0	0	94.45	88.73	93.70	83.33
1	0	93.96	90.60	94.95	83.62
0	1	95.02	88.10	94.44	84.25
0.5	0.5	95.08	88.03	94.04	83.94
0.3	0.7	94.78	88.88	94.67	84.10
0.7	0.3	95.49	88.50	94.47	84.24
1	1	95.89	89.56	94.98	84.37
2	2	95.32	88.45	94.55	83.69

As shown in Table 6, when only the point calculation factor is introduced, i.e., $\lambda =1$ and $\mu =0$, the mAP@50 and mAP@50-95 increase by $1.25\%$ and $0.29\%$, respectively, compared with the conventional IoU-based regression setting $(\lambda =0,\mu =0)$. In addition, this setting yields the highest recall, indicating that the point calculation factor is beneficial for improving localization stability and reducing missed detections. When only the dynamic gradient gain distribution factor is employed, i.e., $\lambda =0$ and $\mu =1$, the model gains $0.74\%$ in mAP@50 and $0.92\%$ in mAP@50-95, which demonstrates that this factor can effectively enhance regression quality by adaptively adjusting gradient allocation during optimization.

Furthermore, when the two factors are jointly introduced, the detection performance is further improved, which confirms that they provide complementary benefits in bounding box optimization. Among all tested settings, the best performance is achieved when $\lambda =1$ and $\mu =1$, where the model obtains $95.89\%$ precision, $89.56\%$ recall, $94.98\%$ mAP@50, and $84.37\%$ mAP@50-95. By contrast, when both hyperparameters are increased to 2, the performance drops noticeably. This phenomenon suggests that excessively strong regression penalties may lead to unstable optimization and oscillation around the optimum, thereby hindering effective convergence.

To further verify the effectiveness of the proposed loss design, we also compare different combinations of classification loss and IoU-based localization loss, and the results are listed in Table 7. Each row in the table corresponds to one specific loss function configuration, where the first column denotes the classification loss and the second column denotes the IoU-based regression loss.

Table 7. Comparison of different classification and IoU loss combinations. Each row corresponds to one loss function configuration.

Classification	IoU	Precision	Recall	mAP@50	mAP@50-95
BCE	GIoU	94.86	88.42	93.96	83.54
BCE	DIoU	93.62	88.24	93.82	83.68
BCE	PowerIoU	93.67	89.02	94.27	83.89
BCE	OIoU	94.45	88.89	94.61	84.15
EMALoss	GIoU	94.61	89.07	94.17	83.94
EMALoss	DIoU	95.44	87.74	94.26	83.92
EMALoss	PowerIoU	95.62	89.11	94.81	84.28
EMALoss	OIoU	95.89	89.56	94.98	84.37

First, the classification loss is fixed as BCE Loss, while different IoU losses are adopted for bounding box regression. The results show that the proposed OIoU loss achieves the best performance among all compared IoU-based losses under the BCE setting, reaching $94.61\%$ mAP@50 and $84.15\%$ mAP@50-95. In particular, compared with the second-best PowerIoU [43], OIoU improves mAP@50 by $0.34\%$ and mAP@50-95 by $0.26\%$, which demonstrates its stronger capability in localization optimization.

Next, when the classification loss is replaced by EMALoss, the model performance is further improved. Among all combinations, the EMALoss + OIoU configuration achieves the best overall results, with precision, recall, mAP@50, and mAP@50-95 reaching $95.89\%$, $89.56\%$, $94.98\%$, and $84.37\%$, respectively. Compared with the second-best EMALoss + PowerIoU combination, it yields additional gains of $0.27\%$ in precision, $0.45\%$ in recall, $0.17\%$ in mAP@50, and $0.09\%$ in mAP@50-95. These results indicate that the proposed OIoU regression loss and the adopted classification loss design can work collaboratively to improve both localization accuracy and overall detection robustness.

4.4.4. Random-Seed Stability Analysis

To further verify that the performance gains are not caused by random fluctuations during training, we conducted additional experiments with different random seeds. Specifically, both YOLOv11 and OShipNet were trained and evaluated three times using seeds 0, 42, and 999, while keeping all other experimental settings unchanged. For each metric, we report the mean, standard deviation, and 95% confidence interval. The 95% confidence interval was computed as

$$C{I}_{95\%}=\overline{x}\pm t_{0.975,n-1}{\displaystyle \frac{s}{\sqrt{n}}}$$

(23)

where $\overline{x}$ denotes the sample mean, s denotes the sample standard deviation, and n is the number of repeated runs.

As shown in Table 8, OShipNet consistently outperforms YOLOv11 across all three random seeds. Averaged over the three runs, OShipNet improves Precision, Recall, mAP@50, and mAP@50-95 by 1.00, 1.50, 1.10, and 2.05 percentage points, respectively. More importantly, the confidence intervals of the two models on the main detection metrics are clearly separated. For mAP@50, the 95% confidence interval is [93.66, 93.82] for YOLOv11 and [94.52, 95.16] for OShipNet. For mAP@50-95, the corresponding intervals are [82.09, 82.43] and [84.10, 84.52], respectively. This indicates that the improvement of OShipNet remains consistent under different random initializations.

Table 8. Performance stability under different random seeds.

Model	Seed/Statistic	Precision	Recall	mAP@50	mAP@50-95
YOLOv11	0	94.27	87.80	93.72	82.29
	42	93.78	88.01	93.78	82.18
	999	94.51	87.95	93.73	82.31
	Mean ± Std.	94.19 ± 0.30	87.92 ± 0.09	93.74 ± 0.03	82.26 ± 0.06
	95% CI	[93.26, 95.11]	[87.65, 88.19]	[93.66, 93.82]	[82.09, 82.43]
OShipNet	0	95.89	89.56	94.98	84.37
	42	94.72	89.92	94.81	84.21
	999	94.97	88.79	94.73	84.34
	Mean ± Std.	95.19 ± 0.50	89.42 ± 0.47	94.84 ± 0.10	84.31 ± 0.07
	95% CI	[93.66, 96.72]	[87.99, 90.86]	[94.52, 95.16]	[84.10, 84.52]

To further provide an intuitive understanding of performance stability, Figure 9 presents boxplot visualizations of the metric distributions across different random seeds. It can be observed that OShipNet consistently achieves higher median performance than YOLOv11 on all evaluation metrics, while maintaining comparable or slightly larger variance due to its higher performance ceiling. More importantly, the interquartile ranges of the two models exhibit minimal overlap, and the overall distributions are clearly separated, especially for mAP@50 and mAP@50-95.

Figure 9. Performance comparison across different random seeds.

4.5. Fine-Tuning Experiment

To verify the generalization capability of our model, we conducted fine-tuning experiments on the Seaship7000 dataset, using weights trained on the MVDD13 dataset as pre-trained initialization parameters. The Seaship7000 dataset was re-split into a fine-tuning training set and a validation set at a 1:1 ratio, with experimental results detailed in Table 9. Overall, the pretrained models showed performance improvements in most metrics compared to their non-pretrained counterparts. In particular, Figure 10 compares the loss and mAP@50-95 of pretrained and nonpretrained OShipNet on the Seaship7000 dataset.

Table 9. Results of fine-tuning experiments on the Seaship7000 dataset.

Model	Pretrained	Precision	Recall	mAP@50	mAP@50-95
Faster-RCNN	NO	89.17	91.93	97.40	71.44
SSD		95.36	91.25	97.52	69.58
YOLOv5		97.46	97.10	98.86	80.30
YOLOv8		97.09	96.81	98.71	80.69
YOLOv11		97.77	96.36	98.83	79.85
YOLOv12		97.40	95.74	98.71	79.53
Swin Transformer		84.12	88.95	97.50	71.50
RT-DETR		95.38	95.59	98.33	79.12
OShipNet (ours)		97.49	97.38	98.89	81.42
Faster-RCNN	Yes	86.17	89.84	97.49	72.87
SSD		95.35	89.47	97.03	70.38
YOLOv5		97.48	97.11	98.94	82.92
YOLOv8		97.89	97.20	98.95	82.65
YOLOv11		97.52	97.33	98.92	83.20
YOLOv12		97.40	97.29	98.81	81.98
Swin Transformer		88.60	92.18	98.08	76.65
RT-DETR		95.92	97.05	98.67	80.16
OShipNet (ours)		97.53	97.89	98.96	84.07

Figure 10. Loss and mAP@50-95 curves of pretrained and non-pretrained OShipNet.

As shown in Figure 10, the pretrained OShipNet achieves lower loss and higher accuracy. Next, our proposed OShipNet achieved the best overall performance after fine-tuning, with Precision of 97.53%, Recall of 97.89%, mAP@50 of 98.96%, and mAP@50-95 of 84.07%, outperforming all other models in each evaluation metric. Compared to the best performing model YOLOv11, OShipNet still leads by 0.04% in mAP@50 and 0.87% in mAP@50-95, demonstrating its improvements in localization capability. OShipNet achieves significant performance superiority over Transformer-based models including Swin Transformer and RT-DETR on both mAP@50 and mAP@50-95 metrics, which validates its strong robustness and generalization capability after pre-training. These results demonstrate that the introduction of pretraining significantly improves the detection performance of OShipNet, further highlighting its strong adaptability and effectiveness in real-world scenarios with different ship types and occlusion conditions.

Finally, we give a visualization of the results of each model on Seaship7000, as shown in Figure 11. Compared to other fine-tuned models, the fine-tuned OShipNet shows more reliable detection performance, accurately locating and identifying targets with high completeness and accuracy. Compared to models such as Faster R-CNN, Swin Transformer, and RT-DETR, the fine-tuned OShipNet produces fewer false positives in the visualized examples. In addition, compared to YOLOv12 and YOLOv8, the fine-tuned OShipNet achieves higher confidence scores for each target while maintaining accurate localization. As a result, the fine-tuned OShipNet yields fewer missed and incorrect detections and shows more consistent alignment with ground truth annotations, especially in complex scenarios with dense or overlapping ship instances. This highlights the strong generalization capability and robustness of OShipNet.

Figure 11. Fine-tuned visualization on Seaship7000 dataset.

4.6. Generalization Experiment

To further evaluate cross-domain occlusion robustness beyond maritime scenes, we conducted additional experiments on the public Crowdhuman dataset. Unlike MVDD13, Crowdhuman is a non-maritime benchmark with substantially different object categories and scene characteristics, but it contains dense layouts and severe occlusion, which makes it suitable for assessing the cross-domain robustness of occlusion-oriented detectors.

The comparative results are shown in Table 10. As observed, the proposed method achieves competitive performance across all evaluation metrics, including Precision, Recall, mAP@50, and mAP@50-95. Specifically, the proposed method attains 87.36%, 76.36%, 85.74%, and 57.35%, respectively, demonstrating consistent improvements over most baseline models. Compared with the strongest baseline (YOLOv12), the proposed method achieves a performance gain of 1.18% in mAP@50-95, indicating better localization accuracy and scale adaptability under varying data distributions. In comparison with two-stage detectors and Transformer-based methods, the proposed approach achieves superior performance while maintaining relatively lower computational complexity. These results indicate that the proposed framework maintains strong feature representation capability and stable detection performance even when applied to datasets with different characteristics.

Table 10. Comparison results on the Crowdhuman dataset.

Model	Precision	Recall	mAP@50	mAP@50-95	FLOPs	Params
Faster R-CNN	79.44	76.71	78.36	46.92	90.9 G	41.35 M
SSD	78.30	62.59	68.20	33.33	137.2 G	24.86 M
YOLOv5	86.53	74.19	84.30	55.21	18.7 G	7.81 M
YOLOv8	86.78	74.76	84.55	55.91	23.4 G	9.83 M
YOLOv11	86.77	74.37	84.50	55.56	21.3 G	9.41 M
YOLOv12	87.18	75.25	85.10	56.17	21.2 G	9.23 M
Swin Transformer	80.25	77.52	78.72	47.42	146.3 G	47.39 M
RT-DETR	85.24	73.05	83.80	52.58	186.2 G	60.90 M
OShipNet (ours)	87.36	76.36	85.74	57.35	29.6 G	14.16 M

Figure 12 illustrates the qualitative comparison of detection results across different models. It can be observed that existing methods exhibit varying degrees of performance degradation when handling complex object distributions and occlusion patterns, including missed detections, inaccurate localization, and redundant predictions. In contrast, the proposed method demonstrates more consistent detection results, with higher alignment between predicted bounding boxes and ground truth annotations, as well as reduced false positives and false negatives.

Figure 12. Crowdhuman Dataset Visualization Results.

Overall, the experimental results demonstrate that the proposed method generalizes effectively across different datasets and maintains robust performance under diverse visual conditions.

5. Discussion

The experimental results indicate that OShipNet is effective for occluded ship detection in complex maritime scenes. The performance gains suggest that the proposed framework is able to better preserve and exploit discriminative information when targets are partially visible, densely distributed, or subject to scale variation. In particular, the improvements observed across the experiments imply that the combination of enhanced feature representation, contextual interaction, and adaptive optimization contributes to more robust detection under challenging conditions.

From the experimental observations, OShipNet shows clear advantages in scenarios where conventional detectors are more easily affected by occlusion and background interference. This indicates that the model is capable of capturing not only local visual cues from incomplete targets but also useful contextual information from surrounding regions, which is important for maritime scenes involving overlapping vessels and cluttered backgrounds. In addition, the stable performance on the Seaship7000 dataset further suggests that the proposed method has a certain degree of generalization ability beyond the primary training setting. Generalization experiments on the Crowdhuman dataset further show that OShipNet can achieve reliable detection performance in more general occlusion scenarios, which provides additional evidence of the generalization capability of the proposed method.

Nevertheless, several limitations remain. First, categories with high visual similarity, such as bulk carriers, cargo ships, and oil tankers, are still prone to misclassification, especially in crowded or heavily occluded scenes where fine-grained structural differences are difficult to preserve. Second, although the proposed framework improves detection accuracy, the introduction of multiple enhanced modules also increases computational cost compared with standard lightweight detectors, which may restrict its use in real-time or resource-limited deployment environments. These findings suggest that future research should place greater emphasis on fine-grained category discrimination and inference efficiency, so as to further improve the practicality of the model in real maritime applications.

6. Conclusions

In this paper, we present OShipNet, an efficient ship detection framework designed to address the challenges of insufficient feature extraction and degraded detection performance under ship occlusion. Specifically, we develop OShipNeXt, a multidomain feature fusion backbone, to enhance semantic representation by exploiting the complementarity between frequency domain and spatial domain features, thereby improving contextual reasoning in occluded regions. In addition, the proposed MSPAM module strengthens feature representation through multiscale processing and adaptive attention refinement, which helps suppress false detections caused by complex backgrounds. Furthermore, a dual-path optimization loss is introduced to jointly improve classification and bounding-box regression, leading to more accurate predictions. Experimental results on the MVDD13 dataset demonstrate that OShipNet achieves 94.98% mAP@50 and 84.37% mAP@50-95, outperforming competing methods in overall detection performance. Further experiments on the Seaship7000 dataset show that OShipNet benefits effectively from pre-training, while results on the Crowdhuman dataset verify its robustness and generalization capability on public occlusion detection benchmarks. Compared with existing methods, OShipNet provides an occlusion-oriented solution by integrating feature representation, contextual refinement, and loss optimization into a unified framework. In future work, we will further investigate lightweight and edge-oriented deployment strategies to better satisfy the stability and efficiency requirements of marine surveillance applications.

Appendix A.1. MVDD13 Dataset

We selected the open dataset MVDD13 [44], which contains occlusion information, as our experimental dataset. MVDD13 comprises 13 categories: Cargo ship, Tanker, Bulker, Containership, Tug, Fishing boat, Drill ship, Passenger ship, Cruise ship, Warship, Sailing boat, Submarine, and Firefighting ship. The entire dataset includes 25,541 training images, 2837 validation images, and 7095 test images. The statistical distribution of the dataset is summarized in Table A1, with the overall sample distribution visualized in Figure A1. The size distribution across vessel categories and per-class instance counts in the testing set are detailed in Figure A2.

Table A1. Distribution of MVDD13 dataset.

Category	Training Set		Validation Set		Testing Set
	Instances	Percentage	Instances	Percentage	Instances	Percentage
Cargo	5522	18.79%	644	19.62%	1473	17.98%
Tanker	3296	11.21%	367	11.18%	926	11.30%
Bulker	3802	12.94%	421	12.82%	1045	12.75%
Containership	781	2.66%	63	1.92%	231	2.82%
Tug	1962	6.68%	216	6.58%	552	6.74%
Fishing	1056	3.59%	94	2.86%	276	3.37%
Drill	938	3.19%	105	3.20%	293	3.58%
Passenger	4132	14.06%	467	14.22%	1127	13.76%
Cruise	3773	12.84%	434	13.22%	1097	13.39%
Warship	300	1.02%	28	0.85%	92	1.12%
Sailingboat	2747	9.35%	330	10.05%	749	9.14%
Submarine	166	0.56%	21	0.64%	63	0.77%
Firefighting	916	3.12%	93	2.83%	269	3.28%

Table A1. Distribution of MVDD13 dataset.

Category	Training Set		Validation Set		Testing Set
	Instances	Percentage	Instances	Percentage	Instances	Percentage
Cargo	5522	18.79%	644	19.62%	1473	17.98%
Tanker	3296	11.21%	367	11.18%	926	11.30%
Bulker	3802	12.94%	421	12.82%	1045	12.75%
Containership	781	2.66%	63	1.92%	231	2.82%
Tug	1962	6.68%	216	6.58%	552	6.74%
Fishing	1056	3.59%	94	2.86%	276	3.37%
Drill	938	3.19%	105	3.20%	293	3.58%
Passenger	4132	14.06%	467	14.22%	1127	13.76%
Cruise	3773	12.84%	434	13.22%	1097	13.39%
Warship	300	1.02%	28	0.85%	92	1.12%
Sailingboat	2747	9.35%	330	10.05%	749	9.14%
Submarine	166	0.56%	21	0.64%	63	0.77%
Firefighting	916	3.12%	93	2.83%	269	3.28%

Figure A1. Dataset distribution of MVDD13 dataset.

Figure A1. Dataset distribution of MVDD13 dataset.

Figure A2. Labels distribution of MVDD test dataset.

Figure A2. Labels distribution of MVDD test dataset.

Appendix A.2. Seaship7000 Dataset

The SeaShip7000 dataset consists of approximately 7000 images collected from real maritime environments. It contains six ship categories, namely bulk cargo carrier, container ship, fishing boat, general cargo ship, ore carrier, and passenger ship. Owing to its diverse imaging conditions and complex background interference, the dataset provides a suitable benchmark for evaluating the robustness and generalization capability of detection models. In this study, the dataset is re-divided into a fine-tuning set and a validation set at a ratio of 1:1, and is further used to assess the cross-dataset generalization performance of the proposed method under both pretrained and non-pretrained settings. The distribution of annotated targets in the SeaShip7000 dataset is illustrated in Figure A3.

Figure A3. Dataset distribution of Seaship7000 dataset.

Figure A3. Dataset distribution of Seaship7000 dataset.

Appendix A.3. Crowdhuman Dataset

The Crowdhuman dataset is further employed to evaluate the robustness of the proposed method in highly crowded and severely occluded scenarios beyond the maritime domain. It contains 19,370 images, comprising 15,000 training images and 4370 validation images. Owing to its dense object distribution and substantial inter-instance occlusion, Crowdhuman serves as a highly challenging benchmark for assessing the model’s capability to perceive and localize targets under complex occlusion patterns. An example image from the Crowdhuman dataset is presented in Figure A4.

Figure A4. Example image from the Crowdhuman dataset.

Figure A4. Example image from the Crowdhuman dataset.