MSC-YOLO: An Accurate and Effective Maritime Ship Detection Model Based on Improved YOLOv11n

Abstract

To address critical challenges in maritime ship detection within complex surveillance imagery, including severe background interference, extreme scale variation, and fine-grained category confusion, this study proposes Maritime Scene Collaborative You Only Look Once (MSC-YOLO), an improved detection model for fixed-location maritime surveillance scenarios. First, a Maritime Scene Adaptive Attention Module (MSAM) is introduced to suppress water-surface clutter and enhance structurally informative ship responses through bidirectional feature regulation, thereby strengthening feature representation in background-complex scenes. In addition, a Scale-aware Dynamic Head (SDA-Head) is designed by integrating deformable convolution with parallel scale-aware prediction branches to improve detection coverage for vessels under pronounced scale variation. Furthermore, a Class Prototype Guided (CPG) module is developed, incorporating class-level prototypes and category-similarity priors to improve the discriminative representation of visually similar ship categories and component states. Experimental results on the constructed maritime surveillance dataset show that MSC-YOLO achieves 0.9723 mAP@50, 0.7315 mAP@50–95, 0.8903 Precision, and 0.9883 Recall. Compared with YOLOv11n, the proposed model improves mAP@50 by 17.77%, Precision by 21.82%, and Recall by 8.16%, indicating clear advantages in target discovery, clutter robustness, and difficult-target coverage in complex maritime surveillance scenes. Visualization and confusion-matrix analyses further show reduced background interference and stronger class-wise discrimination. Overall, MSC-YOLO demonstrates effective and reliable performance for complex maritime surveillance scenarios.

1. Introduction

Ship detection is a fundamental capability in intelligent maritime perception and supports a wide range of applications, including navigation safety in waterways and bridge areas, port scheduling and management, and maritime law enforcement and supervision [1]. Its core task is to accurately localize ships and classify them in surveillance images or videos captured in complex water environments, thereby providing reliable inputs for risk warning, traffic statistics, and behavior analysis [2]. Compared with road or urban surveillance, maritime scenes often contain large water surfaces and strong unstructured interference caused by waves and reflections. They also include complex background structures, such as bridges, shoreline vegetation, and floating objects, which easily lead to false positives and missed detections [3,4]. At the same time, ship scales vary widely under surveillance views. Distant small targets often occupy only a few pixels, and different ship types or ship parts appear similar under low resolution and occlusion, which further increases class confusion [5]. Accordingly, this work focuses on scenario-oriented architectural adaptation of YOLOv11n for fixed-location maritime surveillance. Traditional approaches that rely on manual inspection or simple rules are inefficient and highly subjective, while generic detection models often fail to maintain stable performance under these conditions [6]. There is therefore a practical need for a ship detection method tailored to maritime surveillance that remains robust to complex backgrounds and can better handle the scale variation and category ambiguity commonly observed in fixed monitoring imagery [7].

With the rapid development of deep learning, maritime ship detection has gradually shifted from handcrafted pipelines to CNN-based detectors, among which YOLO-style one-stage frameworks are widely used in maritime vision tasks [8,9]. Recent studies have further extended maritime vision from remote sensing and drone imagery to onboard monitoring and surveillance video, indicating that detector design should be adapted to sensing geometry, target-scale distribution, and application constraints rather than transferred directly from generic object-detection benchmarks [10].

However, most existing studies focus on remote sensing imagery, onboard navigation, or task-specific maritime systems rather than fixed-location sea surveillance with shoreline vegetation, bridge structures, timestamp overlays, and long-range small vessels. Moreover, current detector adaptations usually emphasize either clutter suppression or multi-scale enhancement, while confusion among visually similar ship categories and structural states is modeled less systematically. These limitations motivate a detector that is simultaneously scene-aware, scale-aware, and relation-aware for complex maritime surveillance.

To address these issues, this paper proposes a ship-detection model for complex maritime surveillance scenarios. To compensate for the lack of real-scene data, a dedicated maritime surveillance ship dataset is constructed, and a category-similarity matrix is extracted to characterize prior relations among different ship types and components. On this basis, with YOLOv11n as the baseline, the paper develops an improved framework with three core modules. Specifically, the proposed Maritime Scene Adaptive Attention Module (MSAM) uses a dual-branch mechanism to suppress background clutter, such as wave splashes, and enhance ship-structure features. The proposed Scale-aware Dynamic Head (SDA-Head) combines parallel multi-scale prediction with an auxiliary scale-information branch to strengthen detection responses for distant small targets under pronounced scale variation. In addition, the proposed Class Prototype Guided (CPG) module incorporates prototype regularization with the category-similarity matrix to reduce feature-level confusion among similar classes. The resulting model is designed to improve detection performance and feature discrimination in complex water-surface scenes, with the most pronounced gains concentrated in mAP@50, Precision, and Recall.

2. Related Work

2.1. Maritime Surveillance Datasets and Scenario-Specific Detection Tasks

Recent progress in ship detection has been strongly shaped by the datasets and task settings used for evaluation. Recent reviews of electro-optical and SAR ship detection show that maritime vision research spans optical remote sensing, sea-surface surveillance imagery, infrared observation, and radar-based monitoring, and that each setting exhibits a different combination of background complexity, scale distribution, viewpoint variation, and application constraints [11,12]. Shao et al. [13] addressed obstacle detection for autonomous ship navigation by improving VarifocalNet with deformable convolution, loss redesign, soft non-maximum suppression, and multi-scale prediction so that the detector remained usable under waves, reflections, and sea fog. Gao et al. [14] constructed an infrared small-ship dataset from Bohai Strait videos, used gamma transformation to increase target–background gray contrast, and replaced the YOLOv5m backbone with MobileNetV3 to reduce model size. Folarin et al. [15] investigated vessel identification and classification for controlling a river barrier system, which framed maritime detection as a traffic-management component rather than only an image-benchmark task. Fan et al. [16] further released the MASS-LSVD benchmark to expand first-view marine vessel detection resources with sea-level imagery containing multiple ship and floating-object categories.

Although these studies enrich the maritime detection literature, the task distributions they cover are still not fully aligned with fixed-location sea-surveillance imagery. Remote sensing datasets usually emphasize large field-of-view observation, while first-view navigation datasets focus on shipborne perception and forward collision awareness. Infrared studies place more emphasis on thermal saliency, and barrier-control applications mainly target vessel presence and system triggering. By contrast, fixed monitoring scenes near bridges and shorelines often contain persistent structural background clutter, strong illumination variation, distant small vessels, and fine-grained inter-class ambiguity within the same camera viewpoint. Therefore, the central challenge is not only to build a detector with competitive overall accuracy but also to adapt the representation to a surveillance-specific distribution that differs from standard maritime benchmarks.

2.2. Background-Robust and Scale-Aware Maritime Ship Detection

To improve robustness in cluttered maritime scenes, many studies have enhanced feature extraction by introducing attention mechanisms, context modeling, or prior-guided suppression of background interference. Cui et al. [17] propose a dense attention pyramid network that strengthens informative ship cues and suppresses sea clutter in SAR imagery through attention-enhanced multi-scale features. Gao et al. [18] combine contextual global attention with task-specific context decoupling so that oriented ship detection remains stable in densely distributed and background-complex remote sensing scenes. Wang et al. [19] improve nearshore infrared detection by adding a fusion-attention feature module, SPD-Conv, and Soft-NMS to YOLOv5s, thereby reducing missed detections caused by dense arrangements, partial occlusion, and low-resolution targets. Liu et al. [20] design a joint feature extraction module that models channel-differentiated, context-aware, and global information simultaneously, and they further introduce residual deformable attention and a brightness-distribution auxiliary loss to improve infrared ship localization. Li and Wang [21] embed efficient channel attention into the backbone, combine GELAN with PANet in the neck, and adopt MPDIoU loss so that feature fusion and background suppression are optimized together. Taken together, these studies show that explicit feature modulation is effective for reducing maritime false alarms, but the underlying mechanisms remain tied to their own sensing conditions and target distributions.

Another major line of work addresses extreme scale variation and the frequent misdetection of distant small vessels. Multi-scale feature fusion, additional prediction branches, feature pyramid redesign, and adaptive detection heads are commonly adopted to improve recall on maritime targets of widely varying sizes. Zhang et al. [22] design an anchor-free feature-fusion framework with adaptive bidirectional fusion to strengthen the representation of near-shore tiny ships. Zheng and Liu [23] introduce attention-guided local feature enhancement into the low-level prediction layer and combine it with anchor optimization to reduce missed detections of small ships. Shi et al. [24] incorporate an efficient local attention mechanism into the neck, together with an asymmetric decoupled head and content-aware reassembly upsampling, to improve small-target detection against complex remote-sensing backgrounds. Fang et al. [25] propose YOLO-RSA, which combines a four-layer multi-scale feature pyramid, a small-ship attention mechanism, and a rotated detection head to improve ship detection under large-scale variation and complex optical remote-sensing backgrounds. However, most existing designs still optimize either clutter suppression or scale adaptation as separate objectives, and few detectors are explicitly organized around the combined demands of real surveillance imagery, namely, background suppression, scale variation, and robust small-object recall in one unified architecture.

2.3. Fine-Grained Maritime Target Recognition and Category Relation Modeling

Beyond target localization, maritime surveillance often requires reliable discrimination among visually similar vessel categories and structure-dependent states. This issue becomes especially difficult when targets are distant, partially occluded, or represented by low-resolution features, because category boundaries are then determined by subtle shape cues rather than by large appearance differences. Existing studies on fine-grained ship recognition, therefore, often introduce metric learning, prototype learning, or category-relation constraints to increase inter-class separability. Xu and Lang [26] improve fine-grained SAR ship classification by adding a distribution-shift regularization term to metric learning, which enlarges inter-class margins while preserving intra-class compactness. Zhu et al. [27] further combine lightweight attention with a hybrid metric-learning objective so that the learned embedding remains discriminative even under sidelobe interference and strong visual similarity among ship categories. Seo et al. [28] process airborne hyperspectral data collected from six ports into spectral statistics and RGB representations and show that CNN models outperform conventional classifiers for maritime object identification, again indicating that category-sensitive feature learning is essential when visual evidence is subtle or heterogeneous.

Nevertheless, most of the above studies are developed for classification or identification rather than end-to-end detection. As a result, they do not directly solve the feature ambiguity that emerges when classification and localization must be optimized jointly within a detector. In addition, only limited work explicitly injects prior category relations into the detector feature space, even though maritime categories often possess hierarchical or structural similarity, such as visually related vessel types or different states of the same component. This gap suggests that fine-grained maritime detection can benefit from a mechanism that not only enhances instance features with global category prototypes but also regularizes those prototypes according to known inter-class similarity.

3. Proposed Model

3.1. YOLOv11n Baseline Model

YOLOv11n is a nano-scale, single-stage object detector, and it is adopted here as the baseline architecture [29]. As a compact member of the YOLO detector family, YOLOv11n represents a practical trade-off among model scale, detection accuracy, and computational efficiency. In this study, YOLOv11n was selected as the baseline because the target maritime-surveillance task contains frequent small and distant vessels, substantial background clutter, and a fixed-camera deployment setting, so the model must maintain a relatively low computational burden while still preserving the multi-scale detection capability of the YOLO framework. YOLOv11n provides a suitable starting point for scenario-oriented improvement under this balance of representation capacity and computational efficiency. The basic architecture of YOLOv11n is shown in Figure 1. It consists of a backbone, a neck, and a detection head. For the nano-scale configuration, YOLOv11n contains 182 layers, 2.624 M parameters, and 6.6 GFLOPs, providing a relatively compact computational starting point. In this study, the input image size is $640\times 640\times 3$, and the number of categories is 6. In Figure 1, k denotes the kernel size of the convolution or pooling layer, s denotes the stride, and p denotes the padding size. The backbone starts with two downsampling convolutions with $k=3$, $s=2$, and $p=1$, which generate the $P1/2$ and $P2/4$ stages with 16 and 32 channels, respectively. It then stacks C3k2 blocks and stride-2 convolutions to construct deeper features, producing 64-channel, 128-channel, and 256-channel feature maps at $P3/8$, $P4/16$, and $P5/32$, respectively. At the deepest stage, an SPPF module with a pooling kernel size of 5 aggregates multi-scale context, and a C2PSA module further enhances semantic interaction. The neck follows a PAN-FPN-style fusion path with $\times 2$ nearest-neighbor upsampling and concatenation, while the detection head performs prediction on three scales, namely, $P3/8$, $P4/16$, and $P5/32$, using feature maps with 64, 128, and 256 channels, respectively.

3.2. The Network Structure of MSC-YOLO

Although YOLOv11n performs well on general object detection tasks, there remains considerable room for improvement in detection performance when it is applied to maritime ship detection. As shown in Figure 1, most of the parameters and computation in YOLOv11n are concentrated in the backbone and detection head. At the same time, ship detection in maritime scenes is still challenged by sea-surface clutter, pronounced scale variation, and target orientation diversity, which makes it difficult for the baseline model to learn sufficiently robust and discriminative representations under complex maritime conditions [30]. To address these issues, we propose an improved detector based on YOLOv11n. To reduce the interference caused by complex sea backgrounds and strengthen the representation of ship structures, we introduce a Maritime Scene Adaptive Attention Module (MSAM), which organizes channel-wise background suppression and complementary multi-scale enhancement within a bidirectional regulation framework to refine feature responses. To better handle ships of different sizes, we design a Scale-aware Dynamic Head (SDA-Head) that augments the detection head with scale prediction and parallel detection branches for small, medium, and large targets. To improve discrimination among visually similar ship categories, we further introduce a Class Prototype Guided (CPG) module, which maps features into a prototype space and uses class-level similarity priors to guide feature enhancement. The overall architecture of the proposed model is shown in Figure 2. The three principal inserted components are MSAM in the backbone feature-refinement path, SDA-Head in the scale-aware detection head, and CPG in the prototype-guided discrimination branch. MSC-YOLO is presented as a task-oriented refinement of YOLOv11n for fixed-location maritime surveillance, and the module design is organized around concrete surveillance-scene failure patterns and empirical performance considerations.

3.2.1. Maritime Scene Adaptive Attention Module

To address feature blurring in long-range maritime imaging, where ship targets are easily degraded by sea-surface clutter reflection, fragmented waves, and complex mist occlusion, we propose a maritime surface adaptive attention module. The core design consideration is that maritime surveillance scenes require two coupled operations at the same time: suppressing large-area redundant background responses and enhancing structurally informative ship cues across different receptive fields. Therefore, the proposed MSAM adopts a bidirectional regulation mechanism that jointly organizes background suppression and multi-scale target enhancement, following recent work on attention-enhanced maritime perception and ship detection under haze and low-light degradation [31,32]. In maritime surveillance scenes, background noise often occupies most of the image and follows certain statistical patterns, whereas ship targets exhibit highly structured geometric characteristics. MSAM uses parallel paths to refine and reconstruct the feature flow, with the goal of extracting more discriminative deep features under complex sea conditions. Within this design, one path focuses on suppressing channels strongly associated with background clutter, while the other path focuses on enhancing ship-related structural cues at different receptive-field scales so that the final fused representation is better aligned with the visual characteristics of maritime targets in complex surveillance scenes. In particular, the $3\times 3$ and $5\times 5$ convolutions in the enhancement branch are used as complementary receptive-field operators to capture both fine-grained local details and more complete ship-body structure, while the module-level contribution lies in integrating reverse background-aware suppression and multi-scale enhancement within one bidirectional regulation framework. The structural model of MSAM is shown in Figure 3.

The module first denoises the input wide-area sea-surface features through a background suppression mechanism. Although the sea-surface background changes continuously, it still presents similar frequency characteristics or reflection intensity within a local region. To capture this global environmental context, the module first applies a global adaptive average pooling layer to compress the input feature map X into a global statistical vector and then uses a bottleneck structure with a reduction ratio to learn nonlinear inter-channel dependencies, which is conceptually related to squeeze-and-excitation-style channel recalibration and recent attention-enhanced ship feature modeling in YOLO-based detectors [33]:

$$W_{bs}=\sigma \left(f_{1\times 1}^{up}\left(\delta \left(f_{1\times 1}^{down}\left(GAP\left(X\right)\right)\right)\right)\right)$$

(1)

Here, $W_{bs}\in {\mathbb{R}}^{C\times 1\times 1}$ denotes the generated background suppression weight map, and $X\in {\mathbb{R}}^{C\times H\times W}$ is the input feature map. $GAP$ denotes the global adaptive average pooling operation, which compresses spatial information to capture the global environmental distribution. $f_{1\times 1}^{down}$ and $f_{1\times 1}^{up}$ denote the channel reduction and channel expansion convolution layers, respectively, which together form an excitation structure for extracting high-level semantic features. $\delta$ denotes the ReLU activation function, which improves the nonlinear fitting ability of the network. $\sigma$ denotes the Sigmoid activation function, which normalizes the weights to the range from 0 to 1. After obtaining the weights, the module adopts a reverse weighting strategy as a global-context-guided channel reweighting operation. In this formulation, higher global pooled responses are used to characterize dominant environmental patterns in the scene so that channels contributing more strongly to redundant sea-surface clutter and background interference can be adaptively suppressed:

$$X_{supp}=X\otimes (1-W_{bs})$$

(2)

Here, $X_{supp}$ denotes the feature map after reweighting, and ⊗ denotes element-wise multiplication, where $W_{bs}\in {\mathbb{R}}^{C\times 1\times 1}$ is broadcast along the spatial dimensions to match $X\in {\mathbb{R}}^{C\times H\times W}$. This design attenuates channels associated with globally dominant responses, thereby reducing background-related interference and improving the relative salience of ship-related structures for subsequent feature extraction.

While suppressing background noise, the module also introduces a dual-path feature enhancement mechanism to address the multi-scale geometric variations of ship targets under different sailing angles. Ship targets in images usually consist of salient edge contours and complex superstructures. To balance fine-grained edges and overall structure, the enhancement branch adopts convolution operators with different receptive fields in parallel. Here, the parallel $3\times 3$ and $5\times 5$ convolutions are used as complementary multi-scale feature extractors, and their role is to support the overall bidirectional regulation design rather than to constitute the sole novelty of the module:

$$W_{enh}=Softmax\left(f_{1\times 1}\left([Con{v}_{3\times 3}\left(X\right);Con{v}_{5\times 5}\left(X\right)]\right)\right)$$

(3)

Here, $W_{enh}\in {\mathbb{R}}^{C\times H\times W}$ denotes the joint spatial-channel enhancement weight map. $Con{v}_{3\times 3}$ is a fine-grained convolution operator that extracts high-frequency details such as target edges and masts, while $Con{v}_{5\times 5}$ uses a larger receptive field to capture the integrity of the ship-body structure. $[·;·]$ denotes concatenation along the channel dimension, which provides an initial fusion of geometric information at two scales. $f_{1\times 1}$ serves as an information interaction layer that integrates cross-scale spatial features. In the proposed module, this enhancement branch is used to complement the suppression path with structure-aware reinforcement. As a result, the enhanced representation can preserve local edge information while also strengthening the overall geometric integrity of ship targets within the bidirectional regulation framework. The $Softmax$ operator introduces competition across channels, which enables the model to automatically learn whether edge cues or structural cues should receive higher priority at each pixel. The feature map is then recalibrated with this weight:

$$X_{enh}=X\otimes W_{enh}$$

(4)

Here, $X_{enh}$ denotes the feature representation after multi-scale geometric enhancement, which helps alleviate the problem of blurred ship contours in offshore backgrounds.

Finally, given the dynamic variability of the sea-surface environment, the module adopts an adaptive dynamic fusion strategy to achieve an appropriate balance between noise removal and signal enhancement. By introducing learnable modulation factors, the module automatically adjusts the relative weights of background suppression and feature enhancement according to the complexity of the input image, while a residual connection preserves the integrity of the underlying semantic information:

$$X^{\prime }=\alpha ·X_{supp}+\beta ·X_{enh}+X$$

(5)

Here, $X^{\prime }$ denotes the refined feature map generated by the MSAM module, and $\alpha$ and $\beta$ are learnable parameters initialized to 0.5.

3.2.2. Scale-Aware Dynamic Head

Conventional detection heads usually rely on fixed-size convolution kernels and regular sampling grids. This design makes it difficult for the model to capture the complex boundary characteristics of maritime ships, whose aspect ratios can be highly extreme and whose spatial distributions are often irregular. In addition, standard detection heads lack dynamic scale awareness. When they process both tiny distant vessels and massive nearby cargo ships, the mismatch between the receptive field and the target size often leads to a clear imbalance in detection accuracy. To address these issues, SDA-Head integrates deformable convolution operators with a multi-branch, scale-aware architecture, following the motivation of maritime multi-scale detection in recent YOLO-based ship detectors [34]. The SDA-Head is shown in Figure 4.

To handle the irregular spatial distribution of maritime ship targets, SDA-Head first applies deformable convolution (DCN) to the input feature map for spatial geometric reconstruction [35,36]. This deformable alignment is performed before the feature map is used by any small-, medium-, or large-scale prediction branch. The main idea is to learn additional offsets so that the sampling points of the convolution kernel can shift dynamically according to the actual target contour, thereby removing the restriction imposed by regular convolution on the shape of the receptive field. The computation is defined as follows:

$$X_{deform}=F_{reg}(X,F_{off}\left(X\right))$$

(6)

Here, $X_{deform}$ denotes the shared deformable feature map after geometric reconstruction, and X denotes the original input feature. $F_{off}$ is the offset prediction function, which estimates the displacement of each sampling point relative to the regular grid, while $F_{reg}$ denotes the convolution operator that performs spatial interpolation and feature reconstruction. This shared $X_{deform}$ is first produced by Equation (6) and is then used by the subsequent scale-information extraction and parallel prediction branches. In this way, the model focuses more precisely on key structural regions of ships, such as masts and decks, and improves feature extraction for elongated targets.

Based on the shared geometrically aligned feature $X_{deform}$, SDA-Head further introduces a dedicated scale prediction branch to address the large variation in maritime target size. This branch dynamically estimates the size level of the current target according to the feature distribution and then provides auxiliary scale-aware guidance for the subsequent prediction process. Its core computation is given by:

$$S_{info}=F_{scale}\left(X_{deform}\right)$$

(7)

Here, $S_{info}$ denotes the generated scale-aware feature map, which is computed from the same shared $X_{deform}$ as auxiliary scale-aware guidance for the downstream prediction process, and $F_{scale}$ denotes the convolution sequence used to extract scale distribution features.

After obtaining the shared $X_{deform}$ and the auxiliary scale-aware information $S_{info}$, SDA-Head builds a parallel multi-branch prediction structure to process small, medium, and large targets collaboratively. The small-scale and medium-scale branches mainly use $3\times 3$ convolutions to preserve precise spatial localization, whereas the large-scale branch uses a $5\times 5$ convolution to enlarge the receptive field so that the global structural characteristics of large ships can be covered more completely. The final prediction is obtained by aggregating the outputs of all branches:

$$Out={\displaystyle \frac{1}{3}}\sum _{i\in small,medium,large}{H}_i\left(X_{deform}\right)$$

(8)

Here, $Out$ denotes the final integrated prediction generated by SDA-Head, and $H_{small}$, $H_{medium}$, and $H_{large}$ denote the prediction head branches designed for targets at different scales. All three branches operate in parallel on the shared $X_{deform}$ after the deformable alignment stage in Equation (6), while $S_{info}$ serves only as auxiliary scale-aware guidance in the head. Equation (8) adopts equal-weight aggregation as a lightweight fusion strategy. Since the three branches already perform differentiated scale-specific processing on the shared $X_{deform}$ before fusion, simple averaging provides a direct way to integrate their outputs without introducing additional fusion parameters or routing complexity. Although the large-scale branch uses a larger convolution kernel to enlarge its receptive field within that branch, this architectural difference affects branch-wise feature extraction behavior rather than directly prescribing a larger fusion coefficient at the output level. Therefore, kernel size and output aggregation weight are treated here as two separate design dimensions. Learnable branch weighting or gating remains a meaningful extension for exploring more adaptive branch-fusion strategies under highly heterogeneous object scales.

Finally, SDA-Head outputs predictions that include ship categories and bounding box coordinates based on the shared deformable feature $X_{deform}$, with $S_{info}$ used only as auxiliary scale-aware guidance during the head processing. This collaborative processing mechanism is intended to improve the geometric robustness of the model to complex ship shapes and to reduce the miss rate caused by target-scale variation through complementary features across branches.

3.2.3. Class Prototype Guided Module

In long-range surveillance scenarios, target visual features are often blurred, which makes categories with similar geometric structures, such as cargo ships and passenger ships, prone to confusion. To address this issue, the CPG module introduces learnable global category prototypes as semantic priors and integrates them into instance features, thereby serving to calibrate instantaneous observation features. This design is motivated by prototype-based representation learning and recent feature-enhancement efforts in YOLO-based ship detectors [37,38]. The CPG module is illustrated in Figure 5.

The core idea of this module is to use stable class-level representations to support instance-level feature discrimination. Specifically, the CPG module first uses a feature projection network composed of a linear layer and a ReLU activation function to map the 512-dimensional high-dimensional feature extracted by the backbone network into a 256-dimensional semantic embedding space:

$$z=F_{proj}\left(f\right)$$

(9)

Here, $z\in {\mathbb{R}}^d$ denotes the projected semantic embedding, $f\in {\mathbb{R}}^{512}$ denotes the input high-dimensional feature vector, and $F_{proj}$ denotes the feature projection function.

Within the embedding space, we compute the cosine similarity between the instance embedding z and learned class prototypes P to obtain a similarity vector s, and we aggregate prototypes to form an enhancement vector v:

$$s={\displaystyle \frac{z·P^{\top }}{\left|z\right|\left|P\right|}}$$

(10)

$$v=s·P$$

(11)

We then enhance the original feature with a residual form:

$$f_{enh}=f_{[0:d]}+\alpha ·v$$

(12)

Here, $s\in {\mathbb{R}}^K$ denotes the cosine similarity vector between the instance feature and the K category prototypes, and $P\in {\mathbb{R}}^{K\times d}$ denotes the learnable category prototype matrix. v denotes the semantic enhancement vector obtained by similarity-weighted aggregation, and $f_{enh}$ denotes the final enhanced feature. Here, $f_{[0:d]}$ denotes the first d dimensions of the original feature. In the present implementation, the prototype branch is defined in a d-dimensional embedding space, so the residual fusion is applied to the corresponding d-dimensional slice of the original feature in order to keep the semantic enhancement pathway dimensionally consistent and computationally lightweight. Under this design, the first d dimensions receive the prototype-guided semantic enhancement, whereas the remaining $512-d$ dimensions preserve their original detector representation at this stage. This setting allows the module to inject category-level semantic priors into a controlled part of the representation while avoiding excessive disturbance to the full original feature vector. In addition, $\alpha$ is a learnable scaling factor that controls the contribution of semantic information.

To ensure that the prototypes faithfully reflect the semantic topological structure among categories, the CPG module further introduces a prototype loss constrained by the similarity matrix M during training. M is used only as a training prior and does not access test labels. This constraint aligns the learned normalized prototype-similarity topology with the fixed prior matrix M and sets relation-dependent upper bounds on off-diagonal inter-class prototype similarities:

$$L_{proto}={∥\widehat{P}{\widehat{P}}^{\top }-M∥}_F^2+L_{inter}$$

(13)

Here, $L_{proto}$ denotes the prototype regularization loss, $\widehat{P}{\widehat{P}}^{\top }$ represents the current normalized prototype similarity matrix, and M denotes a fixed target similarity matrix constructed before training from prior category relations. $\widehat{P}$ is obtained by applying row-wise $L_2$ normalization to the learnable prototype matrix P so that the $(i,j)$ entry of $\widehat{P}{\widehat{P}}^{\top }$ is the cosine similarity between the i-th and j-th class prototypes. The norm ${\parallel ·\parallel }_F$ in Equation (13) denotes the Frobenius norm. In the present implementation, M is defined as a symmetric matrix in which the diagonal entries are set to 1.0 to represent complete self-similarity. Off-diagonal entries are assigned according to semantic and structural relatedness among categories: the two bow-mounted conveyor arm state categories are assigned 0.9 because they correspond to the same ship component under different operating states and therefore share highly similar visual structure; the pair Cargo Ship and Passenger Ship is assigned 0.7 because both belong to conventional ship-body categories with relatively similar elongated hull appearance and deck-level structure; other pairs involving ship categories are assigned 0.3 to reflect a weaker but still non-negligible shared maritime-target geometry; and the remaining weakly related category pairs are assigned 0.1 to indicate minimal prior similarity. This matrix is generated once from class names and attribute relations, saved as a precomputed prior, and kept fixed during training rather than learned jointly with the network parameters so that the prototype constraint remains interpretable and stable throughout optimization. The inter-class term $L_{inter}$ is defined as an off-diagonal hinge penalty:

$$L_{inter}={\displaystyle \frac{2}{K(K-1)}}\sum _{1\le i<j\le K}{\left[max\left(0,{\widehat{p}}_i^{\top }{\widehat{p}}_j-M_{ij}\right)\right]}^2,$$

(14)

where ${\widehat{p}}_i$ and ${\widehat{p}}_j$ are the i-th and j-th row vectors of $\widehat{P}$, and $M_{ij}$ is the corresponding prior similarity value. This term penalizes inter-class prototype similarities only when they exceed the prior similarity allowed by M, thereby encouraging lower similarity for weakly related categories while allowing structurally similar categories to retain a higher but bounded similarity. Through this mechanism, the CPG module not only strengthens the classification robustness of instance features with global semantic memory but also improves the discriminative accuracy of MSC-YOLO in multi-class maritime ship detection by explicitly modeling category correlations.

4. Experiment

4.1. Experimental Configuration

The experimental environment and principal hyperparameter settings adopted in this study are summarized in

Table 1. Experimental environment and hyperparameter settings.

Configuration Item	Value
GPU	NVIDIA GeForce RTX 4090 (24 GB)
Programming Language	Python 3.11
Deep Learning Framework	PyTorch 2.1.0 (CUDA 12.1)
Input Image Size	$640\times 640$
Optimizer	SGD
Initial Learning Rate	0.01
Batch Size	32
Training Epochs	300
Momentum	0.937
Weight Decay	0.0005

. All training and evaluation procedures were conducted on a Linux-based computational server running Ubuntu 22.04 LTS. The implementation was developed in Python and executed within the PyTorch framework with CUDA support for GPU-accelerated computation. During optimization, stochastic gradient descent (SGD) was employed, and a warmup strategy was introduced in the initial stage of training to stabilize parameter updates and improve convergence behavior.

4.2. Computational Complexity and Inference Efficiency

To make the deployment cost of the proposed architecture explicit, we further profiled the fully integrated MSC-YOLO model and the YOLOv11n baseline in terms of parameter count, GFLOPs, latency, and FPS. The profiling was conducted on an NVIDIA GeForce RTX 4090 GPU using the same input resolution and batch setting as the main experiments, namely, $640\times 640$ input images and a batch size of 32, with CUDA acceleration, warmup iterations, and timed forward iterations. The results are summarized in

Table 2. Computational complexity and inference efficiency of YOLOv11n and MSC-YOLO.

Model	Parameters (M)	GFLOPs	Latency (ms/Image)	FPS
YOLOv11n	2.6241	6.6143	1.212	824.91
MSC-YOLO	8.7746	15.8471	1.974	506.53

.

As shown in Table 2, the fully integrated MSC-YOLO model increases the parameter count from 2.6241 M to 8.7746 M and the computational cost from 6.6143 GFLOPs to 15.8471 GFLOPs relative to YOLOv11n. This additional cost is primarily associated with the MSAM feature-refinement operations, the scale-aware dynamic head with deformable-alignment operations, and the CPG branch for prototype-guided category discrimination. Under the same RTX 4090 profiling setting, MSC-YOLO achieves 506.53 FPS with an average latency of 1.974 ms per image, while YOLOv11n achieves 824.91 FPS with an average latency of 1.212 ms per image. These results show that the proposed modules introduce a measurable computational overhead, but the resulting inference speed remains practical for GPU-based maritime-surveillance deployment. Thus, MSC-YOLO represents an accuracy-oriented extension of the compact YOLOv11n baseline, with a clear trade-off between stronger maritime-scene representation and increased deployment cost.

4.3. Experimental Datasets

The experimental dataset used in this study was constructed from images acquired under authentic maritime surveillance conditions. The source data consist of 11 groups of sea-surveillance scenes captured from relatively stable viewpoints but with substantial variation in illumination, weather, and background composition. After manual annotation, sample cleaning, and category filtering, six categories were retained for the final detection task, namely, Cargo Ship, Sand Dredger, Mounted Conveyor Arm Lowered (ship with its bow-mounted conveyor arm lowered), Mounted Conveyor Arm Lifted (ship with its bow-mounted conveyor arm lifted), Passenger Ship, and Law Enforcement Vessel. This dataset setting is intended to preserve the practical complexity of real monitoring imagery while focusing the experiments on fine-grained vessel categories and structure-dependent ship states.

The final dataset contains 37,240 images and 79,231 labeled instances. Following a train/validation/test ratio of 7:2:1, the dataset was divided into 26,068 training images, 7448 validation images, and 3724 test images. Before training, the annotations were converted into the YOLO detection format, and conventional data-augmentation strategies were applied to improve robustness to viewpoint perturbation and object-scale variation. Compared with generic object-detection benchmarks, this dataset exhibits more distinctive surveillance characteristics, including fixed-camera viewpoints, persistent shoreline and bridge interference, long-range small vessels, and visually similar fine-grained categories within the same observation domain.

Figure 6 further illustrates the statistical characteristics of the dataset. The class histogram in Figure 6a reveals a clear long-tail distribution, with Cargo Ship, Sand Dredger, and Mounted Conveyor Arm Lowered contributing 30,623, 28,592, and 16,352 instances, respectively, whereas Mounted Conveyor Arm Lifted, Passenger Ship, and Law Enforcement Vessel account for only 1497, 1391, and 776 instances. The resulting maximum-to-minimum class ratio reaches 39.46:1, indicating that category imbalance and limited samples in rare fine-grained classes are non-negligible challenges. This imbalance is closely related to the real maritime-surveillance acquisition scenario, in which different vessel categories and structural states appear with naturally unequal frequencies. During data preparation, manual annotation, sample cleaning, category filtering, and augmentation were jointly used to improve annotation consistency and increase the training diversity of difficult samples. Figure 6b shows that approximately 33.0% of the annotated targets fall into the small-target range, which confirms that distant small vessels constitute a substantial portion of the dataset and makes scale variation a core difficulty. The centroid distribution in Figure 6c yields a spatial uniformity score of 0.802, suggesting that targets are distributed over a relatively broad image region, while still presenting denser aggregation in the main navigation area under fixed-camera observation. In addition, Figure 6d reports mean relative width and height values of 0.1587 and 0.0842, respectively, showing that most targets exhibit elongated bounding-box shapes consistent with the geometric structure of ships. Taken together, these statistics indicate that the dataset simultaneously contains long-tail class imbalance, frequent small objects, surveillance-specific spatial priors, and shape-consistent but fine-grained vessel appearance, thereby providing a challenging benchmark for real-world maritime surveillance detection.

To further illustrate how the training data were enriched before optimization, Figure 7 presents representative composite augmented training samples and their annotations. In these examples, multiple ship instances and background regions are recombined within the same training image, together with visible changes in object scale, rotation, orientation, and spatial position. This visualization reflects how the augmentation strategy increases multi-target composition diversity, strengthens small-target coverage, and preserves annotation consistency under complex maritime backgrounds.

4.4. Evaluation Metrics

To quantitatively evaluate the detection performance of the proposed model, four key metrics are employed: Precision (P), Recall (R), $mA{P}_{50}$, and ${\mathrm{mAP}}_{50:95}$. These metrics (denoted with (B) in the training logs to represent bounding box performance) are defined based on True Positives (TP), False Positives (FP), and False Negatives (FN).

1.

Precision (B): It measures the proportion of correctly predicted positive observations to the total predicted positives. The formula is

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(15)

2.

Recall (B): It represents the proportion of correctly predicted positive observations to all actual positives in the dataset:

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(16)

3.

mAP@50 (B): Mean Average Precision ($mAP$) is the primary metric for object detection. $mA{P}_{50}$ is calculated by averaging the Average Precision (AP) across all categories at an Intersection over Union (IoU) threshold of 0.5. The AP is the area under the Precision–Recall curve:

$$AP={\int }_0^{1}P\left(R\right)dR$$

(17)

$$mA{P}_{50}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}A{P}_i(IoU=0.5)$$

(18)

where N is the total number of classes.

4.

mAP@50–95 (B): This provides a more comprehensive evaluation by averaging the class-wise AP over IoU thresholds ranging from 0.50 to 0.95 with a step of 0.05:

$${\mathrm{mAP}}_{50:95}={\displaystyle \frac{1}{\left|\mathcal{T}\right|}}\sum _{t\in \mathcal{T}}{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}A{P}_i\left(t\right),\mathcal{T}=\{0.50,0.55,\dots ,0.95\},$$

(19)

where $A{P}_i\left(t\right)$ denotes the AP of class i at IoU threshold t, N is the number of classes, and $\left|\mathcal{T}\right|=10$.

Higher values across all these metrics indicate higher detection accuracy and localization precision of the model.

4.4.1. Ablation Experiment

The proposed MSC-YOLO model builds on the YOLOv11n baseline by replacing part of the standard convolutions in the late C2f modules of the backbone with MSAM. Before the classification head, the model inserts the CPG module, which is inspired by prototype learning. On this basis, it further introduces the SDA-Head, which incorporates an adaptive receptive field adjustment mechanism, to replace the original detection head. By integrating sea-surface adaptive attention, dynamic scale perception, and category prototype guidance, the final model forms an enhanced YOLO architecture. To demonstrate more clearly that the proposed enhanced model outperforms the YOLOv11n baseline, this study conducts eight groups of experiments. The results are shown in

Table 3. Results of ablation experiments.

Experiment	MSAM	SDA-Head	CPG	mAP@50	mAP@50–95	Precision	Recall
1				0.7946	0.7238	0.6721	0.9067
2	✓			0.8517	0.7249	0.7832	0.9415
3			✓	0.8824	0.7256	0.8041	0.9528
4		✓		0.9136	0.7275	0.8519	0.9642
5	✓	✓		0.9326	0.7284	0.8635	0.9689
6	✓		✓	0.9418	0.7293	0.8726	0.9713
7		✓	✓	0.9517	0.7302	0.8814	0.9756
8 (Ours)	✓	✓	✓	0.9723	0.7315	0.8903	0.9883

Bold values indicate the best performance for each metric.

.

The ablation results in Table 3 show a clear cumulative improvement trend. Each proposed module improves the baseline, and the full MSC-YOLO configuration achieves the best overall performance when MSAM, SDA-Head, and CPG are combined. Among the single-module variants, SDA-Head delivers the strongest standalone gain, while the pairwise combinations further improve on the corresponding single-module settings, indicating that the three components provide complementary enhancements for maritime target detection.

Another notable pattern is that most gains are concentrated in mAP@50, Precision, and Recall, whereas the improvement on mAP@50–95 remains comparatively limited. From Experiment 1 to Experiment 8, mAP@50 increases by 0.1777, Precision by 0.2182, and Recall by 0.0816, whereas mAP@50–95 rises by 0.0077. Because the baseline backbone, input resolution, training duration, batch size, and early-stopping patience were kept aligned across the ablation settings, the observed differences mainly reflect the effect of the proposed maritime-scene-oriented architectural modules. These modules more directly strengthen target–background separation, small-target coverage, and inter-class discrimination, which first improves whether targets are detected and correctly classified under the IoU = 0.5 criterion. By contrast, mAP@50–95 depends more strongly on fine-grained box-boundary alignment across multiple IoU thresholds. In addition, many targets in the surveillance dataset are small, distant, elongated, or partially occluded, so even slight box-boundary deviations can produce a noticeable penalty when performance is averaged over higher IoU thresholds. Since the proposed modifications mainly strengthen feature representation before final box refinement, while the box-regression loss and label-assignment strategy remain unchanged, the gain on mAP@50–95 is correspondingly more restrained. Overall, the proposed modules contribute most clearly to detection coverage and confidence-related performance, while the improvement in strict localization quality is comparatively moderate.

Figure 8 provides a complementary visualization of the dispersion pattern of the eight ablation settings across mAP@50, mAP@50–95, Precision, and Recall. The full MSC-YOLO configuration still shows the highest central tendency on the four reported metrics, and the cumulative advantage remains stable across the reported metric dimensions.

As shown in Figure 9a, the Recall–Confidence curves indicate that MSC-YOLO maintains high recall over a wide range of low-to-moderate confidence thresholds. The overall recall starts from 0.99 at a confidence threshold of 0.000 and decreases noticeably only at high thresholds, suggesting that most true positives are assigned relatively high confidence scores. However, some class-wise differences remain, as Mounted Conveyor Arm Lowered shows an earlier recall decline than the better-performing categories.

Figure 9b further illustrates the overall detection performance of MSC-YOLO. The all-class Precision–Recall curves are concentrated near the upper-right region, which is consistent with the high mAP@50 value reported in

Table 4. Comparison of detection performance among different algorithms.

Algorithm	mAP@50 (B)	mAP@50–95 (B)	Precision (B)	Recall (B)
Our method	0.9723	0.7315	0.8903	0.9883
YOLOv11n	0.7946	0.7238	0.6721	0.9067
YOLOv8	0.8011	0.6842	0.8662	0.8734
YOLOv9	0.8118	0.6947	0.8326	0.8894
YOLOv10	0.8297	0.7085	0.8613	0.8808
RT-DETR	0.8524	0.7181	0.8746	0.8927
DINO	0.8386	0.7243	0.8821	0.8716
RepGFPN	0.3108	0.2817	0.3066	0.4133
SSD300	0.3025	0.2741	0.2671	0.3233

. The proposed framework further improves difficult-category learning through coordinated suppression of background-dominance bias, enhancement of weak-scale responses, and stronger discrimination among confusing fine-grained classes. Among all categories, Law Enforcement Vessel, Passenger Ship, and Mounted Conveyor Arm Lifted achieve the highest AP of 0.995, while Sand Dredger and Cargo Ship obtain relatively lower AP values of 0.944 and 0.911, indicating that difficult classes still show residual weakness, especially Sand Dredger.

As presented in Figure 9c, precision increases steadily with the confidence threshold and reaches 1.00 at a threshold of 1.000, which suggests that stricter filtering can suppress false positives. Nevertheless, the improvement trend is not fully consistent across categories. Easier classes maintain high precision even at relatively low thresholds, whereas Sand Dredger and Law Enforcement Vessel require higher thresholds to approach their highest reported precision.

According to Figure 9d, a comparatively favorable balance between precision and recall is achieved at a moderately high confidence threshold. The overall F1 score reaches its maximum value of 0.93 at a threshold of 0.698, and the peak region remains relatively broad, indicating that the model is not overly sensitive to small threshold changes near the optimum. Therefore, setting the confidence threshold to around 0.7 may be a reasonable operating point in this setting, although further class-specific calibration may still improve performance on difficult categories.

As shown in Figure 10, the confusion matrix of MSC-YOLO is strongly concentrated along the diagonal, with diagonal values ranging from 0.95 to 0.99, indicating robust class-wise recognition across the six categories. The off-diagonal responses remain low and are mainly confined to a few fine-grained category pairs, which reflects clear inter-class separability under complex maritime scenes. Among them, Sand Dredger shows the broadest error spread, while the two conveyor-arm states remain well distinguished with only slight mutual confusion. Overall, the confusion matrix further supports that MSC-YOLO learns stable responses for both vessel categories and structural-state categories.

4.4.2. Visualization Analysis

Qualitative visualization provides complementary evidence for interpreting the behavior of MSC-YOLO beyond the aggregate metrics reported above. Figure 11 and Figure 12 present representative successful and failed detections under the same frame-level evaluation setting, providing a more complete qualitative view of model behavior in complex surveillance scenes. In the successful examples shown in Figure 11, dense maritime clutter produced by broad water surfaces, shoreline vegetation, bridge structures, timestamp overlays, and visually complex scene composition can be observed, while the predicted boxes remain attached to representative vessel targets rather than drifting toward the surrounding background. These examples also include small, distant, and partially occluded targets as well as multiple vessels in close spatial proximity, yet the detections remain visually aligned with the main target extent. In addition, the model output is not limited to coarse ship-versus-background separation because visually similar categories such as Passenger Ship, Sand Dredger, and Mounted Conveyor Arm Lowered are still distinguished in representative cluttered scenes. Considered together with the quantitative results in Table 4 and the confusion-matrix pattern in Figure 10, these successful cases provide qualitative support for the observed gains in target discovery and fine-grained category discrimination.

Figure 12 presents representative failure cases together with per-image TP, FP, and FN counts, thereby making the residual error structure more explicit. Three recurring difficulty patterns can be observed. First, missed detections still occur for small and distant vessels when target responses are weak relative to large water-surface regions, haze, or low-contrast illumination. Second, some false positives remain near shoreline structures, timestamp overlays, and locally salient background regions, indicating that complex maritime clutter can still trigger ship-like responses. Third, confusion persists for visually similar ship categories when the target occupies only a few pixels or is partially occluded. These failure patterns are consistent with the residual weaknesses already suggested by the class-wise results and confusion-matrix analysis, especially for difficult categories and clutter-sensitive scenes. In contrast, the successful cases in Figure 11 show that MSC-YOLO can still maintain stable multi-target detection and fine-grained category discrimination in many challenging scenes. Taken together, the success–failure comparison indicates that the model performs effectively on difficult maritime surveillance imagery overall, while its remaining weaknesses are concentrated in long-range small targets, severe background interference, and high-similarity category boundaries rather than in ordinary clear-scene targets.

4.4.3. Performance Comparison of MSC-YOLO with Other Mainstream Object Detection Models

To assess the overall performance of MSC-YOLO, this subsection compares it with representative YOLO-series baselines (YOLOv11n, YOLOv8, YOLOv9, and YOLOv10), transformer-based detectors (RT-DETR and DINO), and other mainstream detection models (RepGFPN and SSD300). The quantitative results reported in Table 4 provide the basis for analyzing differences in detection accuracy, localization quality, precision, and recall across these methods.

Table 4 shows that MSC-YOLO achieves the highest values among the compared methods on all four reported metrics. Specifically, MSC-YOLO attains an mAP@50 of 0.9723, exceeding YOLOv9, YOLOv10, RT-DETR, and DINO by 0.1605, 0.1426, 0.1199, and 0.1337, respectively, which indicates a clear advantage under the IoU = 0.5 criterion on the constructed dataset. On the stricter mAP@50–95 metric, MSC-YOLO also ranks first at 0.7315, although the margin becomes comparatively narrow against the strongest competitors, with differences of 0.0072 relative to DINO, 0.0134 relative to RT-DETR, 0.0077 relative to YOLOv11n, 0.0230 relative to YOLOv10, and 0.0368 relative to YOLOv9. This pattern suggests that the most pronounced gains of MSC-YOLO are concentrated in the less stringent detection criterion rather than in uniformly large improvements under higher-IoU evaluation. Such a result is consistent with both the data characteristics and the functional focus of the proposed modules: many targets are small, distant, elongated, or partially occluded, which makes strict IoU-based localization highly sensitive to slight boundary deviations, whereas MSAM, SDA-Head, and CPG more directly improve background suppression, multi-scale target coverage, and category discrimination than the regression mechanism itself. Therefore, the combined design contributes more directly to target discovery and classification stability than to fine-grained box-boundary refinement. In addition, MSC-YOLO achieves the highest Precision (0.8903) and Recall (0.9883), exceeding the strongest competing Precision baseline DINO by 0.0082 and the strongest competing Recall baseline YOLOv11n by 0.0816, respectively. These results indicate that the proposed model reduces false detections while also substantially decreasing missed detections, thereby preserving the strongest overall balance across the four reported metrics. Compared with RepGFPN and SSD300, MSC-YOLO also maintains very large margins on all reported metrics.

At the same time, the comparative advantage of MSC-YOLO is not uniform across all evaluation dimensions. Although MSC-YOLO ranks first on all four reported metrics in Table 4, the gap to the strongest competitors is noticeably smaller on mAP@50–95 than on mAP@50 and Recall. In particular, DINO and RT-DETR approach MSC-YOLO closely on mAP@50–95, which indicates that modern general-purpose detectors remain highly competitive in fine-grained localization quality in this maritime-surveillance scenario. By contrast, the larger gains of MSC-YOLO on mAP@50 and especially on Recall suggest clearer benefits in target discovery, clutter robustness, and difficult-target coverage than in uniformly stronger box-boundary refinement. Moreover, difficult and imbalance-sensitive categories, especially Sand Dredger, still exhibit residual weakness, and the present model does not include a dedicated class-imbalance mitigation mechanism. Overall, these results indicate that MSC-YOLO provides its clearest advantages in complex maritime surveillance scenes where robust target discovery and class discrimination are more critical than uniformly large gains in strict localization quality.

Compared with the baseline methods in Figure 13, MSC-YOLO exhibits the most concentrated diagonal responses and the most compact confusion distribution among the compared methods. The matrix remains predominantly diagonal, and most off-diagonal errors are restricted to only a few minor confusion paths, which indicates comparatively strong class-wise separability. At the same time, Sand Dredger remains the most challenging category, and the class-wise pattern indicates that robustness is still uneven across classes, especially under the severe imbalance described in Section 4.2.

5. Conclusions

To address the core challenges of maritime ship detection in complex surveillance imagery, including water-surface clutter, extreme scale variation, and fine-grained category ambiguity, this paper proposes MSC-YOLO.

Systematic experiments show that the full MSC-YOLO configuration achieves the highest observed performance among the tested variants, while the gain in stricter localization quality remains limited. By integrating MSAM, SDA-Head, and CPG into the YOLOv11n framework, the proposed model achieves 0.9723 mAP@50, 0.7315 mAP@50–95, 0.8903 Precision, and 0.9883 Recall on the experimental dataset, corresponding to improvements of 0.1777, 0.0077, 0.2182, and 0.0816 over the baseline YOLOv11n, respectively. Visualization and confusion-matrix results further suggest reduced interference from complex backgrounds and stronger discrimination among visually similar ship categories and component states. This architecture combines background suppression, scale adaptation, and category-sensitive representation learning through a task-oriented module design.

The relatively smaller mAP@50–95 improvement indicates that the current design contributes more directly to reliable target detection than to further tightening box-boundary regression under stricter IoU thresholds. This result is consistent with the functional focus of the three proposed modules: MSAM reduces clutter interference, SDA-Head improves scale-adaptive feature aggregation, and CPG enhances discrimination among visually similar categories, all of which are more immediately reflected in mAP@50, Precision, and Recall. In addition, the surveillance scenes contain many small, distant, elongated, and partially occluded targets, making strict localization quality particularly sensitive to slight box-boundary deviations. Because the main architectural changes are concentrated in feature refinement and category discrimination rather than in explicit redesign of the regression loss or assignment strategy, the observed gains are larger in detection coverage and category stability than in high-IoU box refinement. The comparative advantage of MSC-YOLO is therefore more pronounced in target discovery, clutter robustness, and difficult-target recall than in uniformly large improvements across all evaluation dimensions. Difficult classes such as Sand Dredger still exhibit residual confusion, and Cargo Ship also remains weaker than the best-performing categories in terms of AP. Future work will consider adding more direct localization-oriented analyses and experiments, including IoU-distribution analysis, localization-error visualization, and stronger regression-oriented optimization, to further evaluate and improve the localization behavior of the model. Overall, the current results indicate that MSC-YOLO provides empirical gains for fine-grained ship detection in fixed-location sea-surveillance imagery within a YOLOv11n-based framework, especially on mAP@50, Precision, and Recall, and may offer a useful reference for maritime visual perception under similar monitoring conditions.