PJ-YOLO: Prior-Knowledge and Joint-Feature-Extraction Based YOLO for Infrared Ship Detection

Abstract

Infrared ship images have low resolution and limited recognizable features, especially for small targets, leading to low accuracy and poor generalization of traditional detection methods. To address this, we design a prior knowledge auxiliary loss for leveraging the unique brightness distribution of infrared ship images, we construct a joint feature extraction module that sufficiently captures context awareness, channel differentiation, and global information, and then we propose a prior-knowledge- and joint-feature-extraction-based YOLO (PJ-YOLO) for use in detecting infrared ships. Additionally, a residual deformable attention module is designed to integrate multi-scale information, enhancing detail capture. Experimental results on the SFISD and InfiRray Ships datasets demonstrate that the proposed PJ-YOLO achieves state-of-the-art detection performance for infrared ship targets. In particular, PJ-YOLO achieves improvements of 1.6%, 5.0%, and 2.8% in mAP₅₀, mAP₇₅, and mAP_50:95 on the SFISD dataset, respectively.

1. Introduction

The safety of maritime navigation is threatened by numerous risk factors, including adverse weather conditions, navigational obstacles, and technical malfunctions, among others. The rapid development of the global shipping industry, coupled with the rise of unmanned surface ships and intelligent industries, has increased the number and variety of ships, making maritime navigation safety more complex [1]. To better mitigate accident risks, ensure navigation safety, and enhance the efficiency of maritime traffic management, effective and accurate detection of ships has become a key research field.

In recent years, with the advancement of infrared technology, infrared object detection has gradually become a research focus. Infrared images, formed by capturing the infrared radiation of objects, possess advantages that are irreplaceable by visible images. These advantages include all-weather capabilities and strong adaptability to complex environments, addressing the limitations of visible images, which are susceptible to environmental factors and struggle with imaging under low-light conditions [2]. Infrared imaging finds widespread applications in various fields such as military, agriculture, medicine and transportation.

Traditional infrared object detection methods include wavelet transform [3], template matching [4], and contrast mechanism-based methods [5], among others. However, these methods have limited accuracy, as they rely on manually designed features. Domain expertise and experience are required to manually select and design features for representing targets. These features often struggle to capture advanced semantic information on complex targets, resulting in poor generalization and limited adaptability to changes in target size. This inadequacy hinders their ability to effectively meet the requirements of ship detection tasks.

The rapid development of deep learning technology has led to powerful feature extraction capabilities, significantly outperforming traditional algorithms in detection performance. This advancement has been widely applied across various fields, such as semantic segmentation [6], keypoint detection [7], and image dehazing [8], further demonstrating its versatility and effectiveness in tackling complex tasks. Convolutional neural network (CNN)-based object detection algorithms excel at automatically learning hierarchical feature representations from raw pixel data, which enables them to detect and localize objects with high accuracy and robustness. Compared to traditional methods, which rely on manually designed features, CNNs can automatically learn and extract complex and abstract features, thereby overcoming the limitations of these methods.

For ship detection, Wen et al. [9] proposed a multi-scale, single-shot detector (MS-SSD) and introduced a scale-aware scheme, which demonstrated excellent performance for small targets while performing less effectively for larger ones. Zhan et al. [10] proposed an edge-guided structure-based object detection network named EGISD-YOLO for infrared ship detection. Li et al. [11] designed a complete YOLO-based ship detection method (CYSDM) for thermal infrared remote sensing ship detection in complex backgrounds, providing a reference for large-scale and all-weather ship detection. Yuan et al. [12] proposed a multitype feature perception and refined network (MFPRN) combining a fast Fourier module and a lightweight multilayer perceptron. This method effectively reduces false detections, but it has a large number of parameters and high computational costs. Although these methods have innovated feature extraction and designed detectors suitable for ship detection, they primarily focus on extracting features from the images without fully utilizing the inherent characteristics of the targets. For example, information such as the shape, motion patterns, and thermal radiation characteristics of the ships is not adequately considered, leading to missed and false detections.

CNN-based detection algorithms have made certain progress in the task of infrared ship detection, but there are still some challenges that need to be overcome. Ship targets in infrared images often exhibit weak features, especially small ones, lacking clear shapes and texture details. The limited recognizable features contribute to increased uncertainty in detection results. Therefore, it is necessary to enhance the extraction of these limited features. Additionally, ships vary greatly in size and have irregular shapes, further limiting the robustness of infrared ship detection. Moreover, the intrinsic characteristics of ship targets can be fully incorporated, enabling more precise and reliable feature extraction tailored to the unique traits of these objects. To address the aforementioned challenges, this paper proposes the PJ-YOLO, which uses YOLOv8 [13] as the baseline. The main contributions of this paper are as follows:

(1)

We propose a novel PJ-YOLO network for infrared ship detection. It integrates prior knowledge auxiliary loss, which leverages the unique brightness distribution characteristics between ships and backgrounds in infrared images as prior knowledge, and also incorporates a joint feature extraction module and a residual deformable attention module.

(2)

To address the challenge of weak and elusive features in infrared images, particularly for small ship targets, we construct a joint feature extraction module to enhance feature extraction ability by utilizing channel-differentiated features, context-aware information, and global information. This module effectively captures subtle characteristics of ship targets.

(3)

Considering the multiscale nature of ship targets, a residual deformable attention module is designed within the higher layers of the backbone network, which effectively integrates multi-scale information to significantly enhance detail capture and improve detection performance.

The remainder of this paper is organized as follows: Section 2 reviews related works. Section 3 describes the overall framework of PJ-YOLO and its modules. Section 4 presents comparative experiments and analyses on the infrared ship datasets. Finally, Section 5 provides the conclusion.

2. Related Work

2.1. Deep Learning-Based Schemes

Deep learning-based object detection methods can be divided into CNN-based methods and transformer-based methods.

CNN-based methods primarily rely on convolution operations to extract features, gradually building an understanding of the image, and can further be categorized into two-stage and one-stage approaches. Two-stage approaches first generate region proposals and then perform further classification and bounding box regression on these proposals. Representative algorithms such as Fast R-CNN [14], Faster R-CNN [15], and Cascade R-CNN [16] have been widely used in object detection tasks. One-stage approaches extract features using CNNs and simultaneously predict the classification and localization of targets, eliminating the step of generating region proposals. Typical algorithms in this category include SSD [17], RetinaNet [18], and YOLO [19]. YOLO, in particular, is renowned for its excellent real-time performance and low computational cost, making it well-suited for real-time applications or scenarios with high speed requirements. For instance, YOLOv8 and YOLOv9 [20] continue the excellent tradition of the YOLO series while introducing advanced techniques and innovations. YOLOv8 adopts a new C2f structure in the backbone network, providing richer gradient flow and employing an anchor-free design. YOLOv9 adopts a lightweight architecture and introduces programmable gradient information, using reversible branches to preserve key deep features for effective object detection. However, for infrared ship detection tasks, these YOLO-like methods still confront significant challenges. Infrared images are characterized by weak and elusive features as well as the presence of numerous small targets that are prone to being missed, which collectively make target detection inherently more difficult. The performance of existing YOLO-like methods remains limited, often resulting in issues such as missed detections and false alarms.

Transformer-based methods leverage the self-attention mechanism to capture global contextual information within images, and RT-DETR [21] is a representative of such approaches. Transformer-based methods excel at capturing global features and long-range dependencies, but they are resource-intensive, slower to train, and often require large datasets for effective performance. By comparison, modern CNN-based methods are generally more suitable for real-time detection tasks.

2.2. Knowledge-Driven Infrared Object Detection

In object detection, “prior knowledge” refers to pre-existing or previously acquired information or assumptions about the target that can help the algorithm better understand and locate it. This prior knowledge can encompass information regarding the target’s size, brightness, shape, appearance characteristics, and spatial distribution.

Prior knowledge plays a crucial role in infrared object detection by leveraging prior understanding of target features and background information, thereby enhancing detection accuracy. Zhu et al. [22] combined the structural prior knowledge of infrared small targets with the autocorrelation of the background to propose infrared small target detection based on low-rank tensor completion and ring top-hat regularization. Liu et al. [23] proposed an infrared object detection algorithm that combines low-rank prior with deep denoising prior. Gao et al. [24] proposed an infrared cirrus cloud detection model based on multi-directional graph learning and local fractal feature prior weight mapping. This model improves the low-rank characteristics and smoothness of the background, resulting in more complete detection of cirrus cloud structures. Zhang et al. [25] proposed a prior model based on the assumption that infrared targets are locally prominent. This model serves as a constraint for the target tensor, effectively reducing the false detection of sparse edge structures. However, existing methods are typically optimized for targets of specific scales, making it difficult to effectively handle multi-scale targets, especially when the target size varies significantly. Additionally, their generalization capability is limited, as they are often designed for specific scenarios and may not perform well in diverse or complex environments.

2.3. Data-Driven Infrared Object Detection

The data-driven approach refers to the methodology of training models with large amounts of annotated data, enabling them to learn and understand the features and contextual information of targets from the data, thereby achieving automated target recognition and localization.

Data-driven deep learning methods have achieved significant application success in various tasks of infrared object detection. To address the challenges posed by the complexity of the background and occlusion among multiple targets, Xu et al. [26] proposed a contour information-guided multiscale feature detection method to enhance the accuracy of pedestrian target detection. Du et al. [27] incorporated the YOLOv4 model into the context of detecting weakly visible and severely occluded vehicles in infrared aerial images amid complex backgrounds. Nonetheless, their method depends on K-means clustering for anchor box determination, necessitating extensive hyperparameter adjustment. To address the challenge of accuracy loss due to the absence of texture features in infrared objects, Bao et al. [28] proposed the Dual-YOLO incorporating attention fusion and shuffle design to achieve simplified and effective fusion of information. However, this solution substantially increases computational demands. To accurately and swiftly detect infrared long-range aerial targets in complex backgrounds, Jiang et al. [29] presented a lightweight multiscale infrared aircraft recognition algorithm. They used a focus module to reduce computational load and a Res2net-based feature extraction design to improve accuracy. Nevertheless, their approach is prone to missed detections. Bhat Kera et al. [30] explored a domain adaptation approach and designed a staged block-wise framework to repurpose the EfficientDet object detector for operation in the thermal infrared domain. Despite its effectiveness, the method faces challenges in managing occlusion scenarios. Zhou et al. [31] integrated existing convolutional backbones with the contemporary transformer architecture to develop a novel combined backbone model named IRMultiFuseNet for infrared ship detection. While the methods outlined above primarily focus on model design, it is also important to consider the inherent characteristics of different types of infrared targets, which could further improve detection performance.

3. Proposed Method

The overall architecture of the proposed PJ-YOLO is shown in Figure 1. In this network, the backbone and neck parts incorporate the joint feature extraction (JFE) module, which enhances feature extraction from multiple dimensions by integrating channel-differentiated features, context-aware information, and global information. This module is particularly effective in capturing subtle characteristics of ship targets, especially in scenarios with weak features or small targets. To better integrate multi-scale information, a residual deformable attention (R-DA) module is introduced in the high layers of the backbone, improving the model’s robustness to scale variations and complex backgrounds. Additionally, considering the significant brightness difference between ships and their backgrounds, an infrared ship prior knowledge auxiliary loss is designed. This loss function utilizes the prior knowledge of brightness distribution differences to guide the network in accurately locating targets, effectively reducing false alarms and missed detections. By combining the JFE module, R-DA module, and prior knowledge auxiliary loss, PJ-YOLO achieves an optimal balance between accuracy and speed, demonstrating strong potential for practical applications in infrared ship detection tasks.

3.1. Joint Feature Extraction Module

In infrared ship images, the available target features are limited, and it is particularly difficult to identify and obtain relevant features for small infrared targets. Therefore, a joint feature extraction (JFE) module has been designed to enhance feature extraction capability across multiple dimensions, including channel, context-aware, and global information. The structure of the JFE module is shown in Figure 2. The number of channels in the feature map is adjusted through a 1 × 1 convolution followed by multi-scale and global feature extraction using Multi-Scale Feature Extractor (MSFE) and Global Feature Extractor (GFE), respectively, with subsequent feature fusion. The residual connection designed after the MSFE facilitates the flow of information between network layers.

The MSFE consists of two branches, each designed to enhance the network’s focus on differentiated features and better capture context-aware information. The first branch is implemented with Self-Calibrated Convolution (SCConv) [32]. Similar to regular convolutions, SCConv separates each channel of the convolution. However, SCConv channels are responsible for specific functions or information. Through dual mapping in the original scale space features and the downsampling latent space features, the network focuses on differentiated features, enhancing channel feature extraction capabilities. The second branch is achieved through three dilated convolutions with different dilation rates. By introducing dilation into the convolutional kernels, the receptive field is expanded, allowing for better capture of context information and strengthening the extraction of spatial features. Compared to regular convolutions, dilated convolutions maintain the same receptive field size while reducing the number of parameters and computational load. Finally, the features extracted from both branches are fused. This process can be expressed as follows:

$$\mathrm{MSFE}\left(x\right)=\mathrm{ReLU}\left(\mathrm{BN}\left(\mathrm{Concat}\left(\mathrm{SCConv}\left(x\right),{\mathrm{DConv}}_1\left(x\right),{\mathrm{DConv}}_3\left(x\right),{\mathrm{DConv}}_5\left(x\right)\right)\right)\right),$$

(1)

where x is the input feature, and $DCon{v}_1$, $DCon{v}_3$, and $DCon{v}_5$ represent dilated convolutions with dilated coefficients of 1, 3, and 5, respectively.

The GFE first processes the input by applying ReLU activation subsequent to both global max-pooling and global average-pooling operations. Global max-pooling captures the most prominent features by selecting the maximum value from each feature map, effectively highlighting the most significant characteristics of the image. In contrast, global average-pooling computes the average value of each feature map, maintaining the overall distribution of features and retaining the global context of the image. Finally, the activated features are processed by a fully connected layer, which integrates the salient and global information, thereby guiding the fusion of multi-scale features and improving the overall feature representation. This procedure can be represented as follows:

$$\begin{array}{cc}\hfill \mathrm{GFE}\left(x\right)& =\mathrm{Sigmoid}\left(\mathrm{FC}\right(\mathrm{ReLU}\left(\mathrm{AvgPool}\right(x\left)\right)\left)\right)\hfill \\ \hfill & +\mathrm{Sigmoid}\left(\mathrm{FC}\right(\mathrm{ReLU}\left(\mathrm{MaxPool}\right(x\left)\right)\left)\right)\hfill \end{array},$$

(2)

where x is the input feature.

3.2. Residual Deformable Attention Module

To enhance the model’s focus on objects of varying scales and improve the network’s perception of critical features, an attention module is designed in the high layers of the backbone network. This aims to better capture the features of the targets.

The structure of the residual deformable attention (R-DA) module is shown in Figure 3. The input features first pass through 1 × 1 convolution for cross-channel aggregation. Then, RepConv [33] reparameterization is applied to enhance inference speed without degrading accuracy. Finally, the features pass through DAttention [34], with a residual connection introduced. The inclusion of the residual connection helps retain the original feature information, preventing the loss of important details and semantic information during feature transmission. This makes the network easier to train and addresses potential issues of information loss and gradient vanishing in attention modules. By establishing a connection between the attention module and the residual connection, the network can more accurately learn and utilize key features.

The main motivation of RepConv is to transform as many convolutions as possible into 3 × 3 convolutions, aiming to improve inference speed. As shown in Figure 3, the simplification process of RepConv involves unifying convolutions of different sizes into a single 3 × 3 convolution, followed by an equivalent merging of the batch normalization (BN) parameters into the 3 × 3 convolution. This equivalent merging method converts the mean and variance of BN into the weights and biases of the convolution, allowing for a single convolution operation during inference and thereby improving inference efficiency.

Inspired by deformable convolutions, the DAttention focuses only on a set of key sampling points around a reference point, rather than processing all pixels in the image. This significantly reduces computational overhead while maintaining good performance. Given the input feature map, reference points are generated. To obtain the offset for each reference point, the feature map is linearly projected to the query token. The offset is generated by the offset network, and it is applied to the reference points. In the offset network, the input features are first captured by deep convolution to extract local features. Then, a GeLU activation function and a 1 × 1 convolution are applied to obtain the two-dimensional offset. After obtaining the reference points, local features are captured through bilinear interpolation and used as the input for both key k and value v. Subsequently, the operations of q, k, and v follow the standard self-attention, with an additional relative position offset compensation. The process can be formulated as follows:

$$q=x{W}_q,\tilde{k}=\tilde{x}{W}_k,\tilde{v}=\tilde{x}{W}_{\nu },$$

(3)

$$\Delta p={\theta }_{offset}\left(q\right),$$

(4)

$$\tilde{x}=\varphi (x;p+\Delta p),$$

(5)

where x and $\tilde{x}$ represent input and sample features, $\tilde{k}$ and $\tilde{v}$ represent deformable key embeddings and value embeddings respectively, and $\varphi$ represents bilinear interpolation operation.

3.3. Loss Function

In object detection, the role of the loss function is to guide the optimization process by quantifying the difference between the model’s predictions and the ground truth, with the goal of enabling the model to accurately and robustly detect targets. Regression loss is a crucial component of object detection, as it guides the model in adjusting the predicted bounding boxes during training to more accurately locate the targets. We designed the Prior Knowledge Auxiliary Loss (PKA Loss), incorporating prior knowledge of infrared ships to guide the model in more accurately localizing ship targets based on brightness differences.

3.3.1. Loss Function of the Regression Branch

In vanilla YOLOv8, the regression branch includes the Distribution Focal Loss (DFL) and CIoU Loss functions. The DFL enables the network to quickly focus on the distribution of positions close to the target location, while the CIoU Loss improves upon the IoU Loss by enhancing the precision of the bounding box regression. The DFL is defined as follows:

$$L_{DFL}^i=-((y_{i+1}-y)log\left(S_i\right)+(y-y_i)log\left(S_{i+1}\right)),$$

(6)

where $y_i$ and $y_{i+1}$ present the values approaching the continuous label y from the left and right sides, and $S_i$ represents the Sigmoid output of the i-th branch.

Compared to the IoU loss function, CIoU incorporates considerations of aspect ratio and center point distance in the penalty term. This allows for better adaptation to targets of varying scales and more accurately measures the overlap between the predicted bounding box and the ground truth bounding box. The definition of CIoU is as follows:

$$IoU={\displaystyle \frac{B_{gt}\cap B_{pred}}{B_{gt}\cup B_{pred}}},$$

(7)

$$v={\displaystyle \frac{4}{{\pi }^2}}{\left(arctan{\displaystyle \frac{w^{gt}}{h^{gt}}}-arctan{\displaystyle \frac{w}{h}}\right)}^2,$$

(8)

$$\alpha ={\displaystyle \frac{v}{1-IoU+v}},$$

(9)

$$L_{CIoU}=1-IoU+\alpha \nu +{\displaystyle \frac{{\rho }^2\left(b,b^{gt}\right)}{c^2}},$$

(10)

where $B_{gt}$ denotes ground truth bounding box, $B_{pred}$ denotes predicted bounding box, v is used to measure the scale consistency between two bounding boxes, and $\alpha$ is the weighting coefficient. b and $b^{gt}$ are predicted and ground truth bounding box centers, ${\rho }^2(·)$ represents Euclidean distance between the centers of the two boxes, and c represents the diagonal distance of the smallest enclosing area that can simultaneously contain both boxes.

3.3.2. Prior Knowledge Auxiliary Loss

In nature, any object with a temperature above absolute zero (0 K) emits infrared radiation. Infrared images are formed by capturing the thermal radiation emitted by objects, resulting in different brightness levels corresponding to temperature differences. As shown in Figure 4, the three scenarios all demonstrate a clear difference in brightness levels between the ship targets and their backgrounds. Ships in infrared images typically have higher temperatures compared to their background, such as water surfaces, making them appear brighter in infrared images. This characteristic can be leveraged to assist in object detection.

As shown in Figure 5, the mean brightness of each target bounding box in the infrared ship image is displayed. It can be observed that there is a noticeable difference in the mean brightness between the ground truth (GT) bounding box and the predicted bounding box. Specifically, for ship (c), the brightness difference between the two boxes reaches 3.15. This significant difference in mean brightness indicates that the predicted bounding box does not fully align with the target, suggesting that some portions of the ship’s main body are not accurately captured within the predicted box. This misalignment leads to the observed discrepancy in average brightness. To address this issue, Prior Knowledge Auxiliary (PKA) Loss was designed by incorporating the brightness differences between the GT and predicted bounding boxes of infrared ships as prior knowledge. The PKA Loss is combined with the CIoU from vanilla YOLOv8 to form a new regression loss, named CPA Loss. The values obtained by normalizing the pixel values of the grayscale image to a range of 0 to 1 represent brightness. There is a significant difference in brightness distribution between the target and the background. This difference in brightness distribution is represented by the mean brightness. To ensure an accurate calculation of the mean brightness with an equal number of pixels in both target boxes, the width and height of the boxes are adjusted to be identical. As shown in Figure 6, the green box represents the original bounding box, and the red box represents the predicted bounding box. By fixing the center points of the target boxes, the width and height are adjusted to the minimum values of the width and height of the two boxes, respectively. The pseudo-code of PKA Loss is shown in Algorithm 1.

Algorithm 1: Pseudo-code of PKA Loss

The new regression loss, CPA Loss, is obtained by combining CIoU with the PKA Loss at different weight ratios:

$$L_{PKA}={\displaystyle \frac{{\sum }_{i=1}^{|{\mathcal{B}}_p^{\prime }|}{\sum }_{j=1}^{N_i}{\parallel P_{pi}\left(j\right)-P_{ti}\left(j\right)\parallel }_1}{|{\mathcal{B}}_p^{\prime }|}},$$

(11)

$$L_{CPA}=\omega L_{CIoU}+(1-\omega )L_{PKA},$$

(12)

where $L_{PKA}$ represents the PKA Loss value, $P_{pi}\left(j\right)$ is the brightness value of the predicted bounding box, $P_{ti}\left(j\right)$ is the brightness value of the original bounding box, $\parallel P_{pi}\left(j\right)-P_{ti}{\left(j\right)\parallel }_1$ is the pixel-level difference, $N_i$ is the number of pixels in each bounding box, ${\mathcal{B}}_p^{\prime }$ is the number of pixels in each bounding box, $\omega$ is the loss weight ratio, and $L_{CIoU}$ is the CIoU Loss value.

4. Experimental Results

4.1. Experimental Settings

The operating system used in this experiment is Ubuntu, with an Intel Core i9 10920x CPU and an Nvidia GeForce RTX 4090 GPU. The deep learning framework employed is PyTorch-GPU 2.1.1, with CUDA version 12.1. The programming language used is Python 3.8.

The number of training epochs is set to 300, with a batch size of 16. The warm-up batch size is set to 5. The optimizer used is SGD with a momentum of 0.937. The initial learning rate is 0.01, and the learning rate decay strategy employs the cosine annealing algorithm.

To enrich the background and increase the quantity of small objects, mosaic data augmentation is employed during the training process. This involves randomly cropping, scaling, and arranging four images and then concatenating them into a single image.

4.2. Datasets

The SFISD dataset [35] consists of 1070 infrared ship images, including single targets, multiple targets, dense targets, and overlapping targets. The dataset comprises 749 images for training and 321 images for testing. The InfiRray Ships dataset (www.iraytek.com (accessed on 15 June 2024)) contains a total of 8402 images of seven classes of infrared ships captured in various scenarios, with 6721 images for training and 1681 images for testing. As shown in Figure 7, the width and height of infrared ship images were normalized to analyze the scale distribution of ship targets. This normalization process involved dividing the width of the bounding box by the image width and the height of the bounding box by the image height, resulting in normalized width and height values ranging between 0 and 1. Statistical analysis of these normalized values revealed the presence of multi-scale ship targets in both datasets, with a significant proportion of small targets. The experiments in this paper were based on the two datasets mentioned above.

4.3. Evaluation Metrics

To assess the performance of the object detection model, standard evaluation metrics, such as mean value of average precision (mAP), parameters, and floating-point operations per second, are used in this study. AP (average precision) refers to the area under the precision–recall (P-R) curve, with recall as the horizontal axis and precision as the vertical axis. The calculation is expressed as Equation (13):

$$\begin{array}{cc}\hfill AP& ={\int }_0^{1}p\left(r\right)dr,\hfill \end{array}$$

(13)

where p represents precision, and r represents recall.

To provide a more detailed evaluation of the model’s performance across different object sizes, the metrics AP_S, AP_M, and AP_L are used. AP_S is the AP value for detecting small objects with pixel sizes less than 32 × 32, while AP_L is the AP value for detecting large objects with pixel sizes greater than 96 × 96. Additionally, mAP₅₀ and mAP₇₅ represent the average precision at IoU thresholds of 0.50 and 0.75, respectively. mAP_50:95 refers to the mean average precision calculated over 10 mAP values, where the IoU threshold ranges from 0.50 to 0.95 with intervals of 0.05. The mAP is the mean of the AP values for each category, and the calculation is expressed as Equation (14):

$$mAP={\displaystyle \frac{{\mathsf{\Sigma }}_{i=1}^{C}A{P}_i}{c}},$$

(14)

where c represents the number of classes, and $A{P}_i$ represents the mean average precision for the i-th category.

4.4. Results and Analysis

4.4.1. Quantitative Results

To validate the infrared ship detection performance of the proposed PJ-YOLO, we compared it with nine state-of-the-art methods, including FCOS [36], CenterNet [37], TOOD [38], Sparse R-CNN [39], MFPRN [12], RT-DETR-r18 [21], YOLOv8 [13], YOLOv9 [20], and YOLOv10 [40]. RT-DETR-r18 refers to using ResNet18 as the backbone. The YOLO series networks are all compared using versions with a similar scale to the proposed PJ-YOLO. The proposed method was separately trained and tested on the SFISD dataset and the InfiRray Ships dataset. Our method outperforms the others based on the experimental results from both datasets.

(1)

Experiments on the SFISD dataset

The experimental comparison results on the SFISD dataset are shown in

Table 1. Performance of different methods on the SFISD dataset.

Method	Year	mAP50	mAP75	mAP50:95	APS	APM	APL	Params (M)	FLOPs (G)
FCOS [36]	2019	72.6	51.5	48.3	6.8	44.8	74.5	31.8	51.5
CenterNet [37]	2019	88.1	52.5	51.3	12.5	46.0	72.4	32.7	109.7
TOOD [38]	2021	79.3	50.6	47.5	7.3	44.1	73.8	32.0	74.4
Sparse R-CNN [39]	2023	83.2	47.4	47.5	14.0	44.8	74.0	106.0	43.0
MFPRN [12]	2023	84.0	56.5	52.6	10.2	50.7	76.7	64.6	178.0
RT-DETR-r18 [21]	2024	92.8	67.9	61.3	19.6	55.3	77.9	20.0	60.0
YOLOv8 [13]	2023	92.0	69.2	63.0	17.6	57.2	80.3	11.2	28.6
YOLOv9 [20]	2024	91.9	68.8	62.3	13.1	57.7	80.2	7.1	26.4
YOLOv10 [40]	2024	92.1	68.7	61.6	19.6	55.7	78.9	7.2	21.6
PJ-YOLO		93.6	74.2	65.8	22.6	58.3	80.5	15.2	35.3

Note: Bold value indicates the best results on the evaluation metrics.

. Compared to the baseline method, the proposed PJ-YOLO demonstrates improvements of 1.6%, 5.0%, and 2.8% in mAP₅₀, mAP₇₅, and mAP_50:95, respectively. Notably, PJ-YOLO achieves a remarkable 5.0% improvement in AP_S for small object detection and maintains superior performance across various target sizes.

(2)

Experiments on the InfiRray Ships dataset

The experimental comparison results on the InfiRray Ships dataset are shown in

Table 2. Performance of different methods on the InfiRray Ships dataset.

Method	Year	mAP50	mAP75	mAP50:95	APS	APM	APL	Params (M)	FLOPs (G)
FCOS [36]	2019	84.2	58.1	54.5	32.4	56.8	68.6	31.8	51.5
CenterNet [37]	2019	87.3	59.1	55.6	31.6	56.9	68.1	32.7	109.7
TOOD [38]	2021	85.3	60.2	56.7	34.7	58.1	70.7	32.0	74.4
Sparse R-CNN [39]	2023	90.7	61.2	59.0	34.7	40.2	72.2	106.0	43.0
MFPRN [12]	2023	89.6	64.7	60.4	39.1	61.8	72.4	64.6	178.0
RT-DETR-r18 [21]	2024	93.2	69.1	64.3	44.4	65.4	76.9	20.0	60.0
YOLOv8 [13]	2023	92.6	69.8	65.3	43.8	63.0	76.7	11.2	28.6
YOLOv9 [20]	2024	91.5	66.8	63.1	41.6	63.0	75.6	7.1	26.4
YOLOv10 [40]	2024	92.5	69.6	64.6	43.5	63.3	74.8	7.2	21.6
PJ-YOLO		93.6	71.9	67.4	45.6	64.8	78.7	15.2	35.3

Note: Bold value indicates the best results on the evaluation metrics.

. Compared to the baseline method, the proposed PJ-YOLO demonstrates improvements of 1.0%, 2.1%, and 2.1% in mAP₅₀, mAP₇₅, and mAP_50:95, respectively. The detailed category-wise comparison results are presented in

Table 3. Performance of different methods on specific categories on the InfiRray Ships dataset.

Method	AP50:95
	All	Liner	Bulk	Warship	Sailboat	Canoe	Container	Fishing
FCOS [36]	54.5	47.2	72.6	74.1	44.1	37.3	74.1	32.2
CenterNet [37]	55.6	47.1	72.7	75.6	46.0	40.3	74.7	32.6
TOOD [38]	56.7	46.5	75.7	77.2	46.3	38.3	76.3	36.8
Sparse R-CNN [39]	59.0	47.9	76.6	78.4	48.0	41.2	78.4	42.5
MFPRN [12]	60.4	49.4	76.4	79.8	50.3	42.7	79.4	44.7
RT-DETR-r18 [21]	64.3	55.5	79.8	81.2	55.7	49.8	80.2	48.0
YOLOv8 [13]	65.3	59.0	82.5	83.7	57.8	51.5	75.6	46.8
YOLOv9 [20]	63.1	56.1	79.1	80.9	56.7	49.6	76.3	43.4
YOLOv10 [40]	64.6	59.2	79.2	82.3	56.8	50.7	77.8	46.4
PJ-YOLO	67.4	61.0	83.5	85.4	59.1	54.0	80.3	48.8

Note: The results in bold font and underlined indicate the best and suboptimal results on the evaluation metric, respectively.

. It can be seen that compared to the SOTA methods, the proposed PJ-YOLO shows significant improvements in metrics for each category of ship targets. Specifically, for individual categories, PJ-YOLO achieves the best results in all categories.

4.4.2. Qualitative Results

The visual comparison results on the SFISD dataset are shown in Figure 8. From left to right, the figures display the ground truth (GT) and the visualization results of YOLOv8, YOLOv9, YOLOv10, and PJ-YOLO, respectively. In Figure 8a, YOLOv9 mistakenly identifies the background on the left as a ship target, and YOLOv10 fails to detect the small target in the center. In Figure 8b, YOLOv9 fails to detect the ship target on the left, while YOLOv10 mistakenly identifies the shoreline as a ship target. In Figure 8c, both YOLOv8 and YOLOv9 mistakenly identify the background as ship targets, and YOLOv9 and YOLOv10 fail to detect the small target on the right. In Figure 8d,e, YOLOv8, YOLOv9, and YOLOv10 exhibit missed detections for small targets. The proposed PJ-YOLO effectively addresses the issues of missed detections and false positives prevalent in other methods, demonstrating superior performance in detecting infrared ship targets across various scales.

The visual comparison results on the InfiRray Ships dataset are shown in Figure 9. In Figure 9a, YOLOv8 and YOLOv10 fail to detect the target, and YOLOv9 misidentifies a canoe as a warship. In contrast, the proposed PJ-YOLO accurately detects the target. In Figure 9b, YOLOv9 and YOLOv10 misclassify the bulk carrier as another category. In Figure 9c, YOLOv8 and YOLOv10 fail to detect the ships that are close to the shore. Figure 9d depicts a relatively blurry scene. In Figure 9d, YOLOv8, YOLOv9, and YOLOv10 all exhibit false detections, incorrectly identifying shore objects and structures as ships, whereas the proposed PJ-YOLO accurately distinguishes between these objects and true ship targets. In Figure 9e, other methods exhibit numerous instances of missed detections, whereas the proposed PJ-YOLO accurately identifies all the targets. Through comparisons in diverse scenarios, the superior performance and robustness of our proposed method are clearly illustrated.

Grad-CAM [41] is a technique that highlights the regions in the input image that are the most important for the model’s prediction. Visualization experiments using Grad-CAM were conducted on the baseline method and the proposed PJ-YOLO. By utilizing the gradient information from the feature map, the corresponding weights were generated to obtain heatmaps. The comparison of image heatmaps across different stages is shown in Figure 10. It can be observed that the class activation mapping of PJ-YOLO focuses more on the target area, while the baseline method exhibits a scattered attention pattern, focusing not only on the target but also on irrelevant regions such as the shore and non-ship areas on the water surface. Guided by prior knowledge of infrared ships, the proposed JFE module enhances the network’s feature extraction capability, enabling it to concentrate more on the target area of infrared ships.

4.5. Ablation Study

To better explore the detection performance of the PJ-YOLO proposed in this paper and verify the efficacy of each module, we conducted ablation experiments on the SFISD dataset. Ablation experiment comparisons are shown in

Table 4. Ablation experiment of each module.

Method	JFE	R-DA	PKA Loss	mAP50	mAP75	mAP50:95	Params (M)	FLOPs (G)
Baseline	-	-	-	92.0	69.2	63.0	11.2	28.6
1	√	-	-	92.7	70.4	63.6	15.0	35.2
2	√	√	-	93.1	71.0	64.5	15.2	35.3
3	√	-	√	93.4	72.4	64.6	15.0	35.2
4	√	√	√	93.6	74.2	65.8	15.2	35.3

. The JFE module improves the model’s mAP₅₀ by 0.7%, mAP₇₅ by 1.2%, and the mAP_50:95 by 0.6%. Introducing the R-DA attention module further increases the mAP₅₀ by 0.4%, mAP₇₅ by 0.6%, and the mAP_50:95 by 0.9%. Additionally, the introduction of PKA Loss brings about further improvements in performance. However, with the model improvements, the parameters and computational complexity of the model also increase.

The visualization results of the ablation study are shown in Figure 11. By comparing the performance of the baseline model with the incremental addition of the JFE module, R-DA attention module, and PKA Loss, the figure illustrates the performance improvements contributed by each module to the overall model. It can be observed that the issues of missed detections and false detections have been improved and resolved.

To evaluate the effectiveness of the CPA Loss, we specifically designed a series of ablation experiments on the SFISD dataset, where comparisons were performed between CPA Loss and other loss functions including GIoU [42], DIoU [43], SIoU [44], EIoU [45], and CIoU [43]. The experimental results, shown in

Table 5. Ablation study on the comparison of different loss functions.

Loss	mAP50	mAP75	mAP50:95
GIoU [42]	93.1	70.3	63.7
DIoU [43]	93.0	71.2	63.9
SIoU [44]	92.9	71.1	64.2
EIoU [45]	92.9	71.0	63.3
CIoU [43]	93.1	71.0	64.5
CPA	93.6	74.2	65.8

, clearly demonstrate that CPA Loss outperforms other loss functions in the task of infrared ship detection and has a positive impact on improving the overall performance of the model.

To investigate the impact of different weight ratios between CIoU Loss and PKA Loss in the regression loss component, five sets of ablation experiments were designed on the SFISD dataset. The comparison results are shown in

Table 6. Ablation experiment of weight ratios.

${\mathit{\omega }}_{\mathit{CIoU}}$	${\mathit{\omega }}_{\mathit{PKA}}$	mAP50	mAP75	mAP50:95
1	0	93.1	71.0	64.5
0.8	0.2	93.2	72.6	65.1
0.6	0.4	93.6	74.2	65.8
0.4	0.6	93.9	72.3	64.4
0.2	0.8	93.4	71.8	65.0

.

4.6. Computational Complexity

To evaluate the computational complexity of the proposed PJ-YOLO, a comparative experiment was conducted with the batch size set to 1 to simulate real-world scenarios. Figure 12 demonstrates the computational complexity comparisons between PJ-YOLO and other competing methods. Initially, the computational complexity of different methods is evaluated based on latency and floating-point operations (FLOPs), and the performance of these methods is evaluated using the SFISD dataset.

The real-time performance comparison results are presented in

Table 7. Comparison of FPS and latency for different methods.

Method	RT-DETR-r18	YOLOv8	YOLOv9	YOLOv10	PJ-YOLO
FPS	49.0	158.7	67.1	133.3	87.7
Latency (ms)	20.4	6.3	14.9	7.5	11.4

. The proposed PJ-YOLO achieves 87.7 FPS. As illustrated in Table 7, while its FPS is marginally lower and its latency is somewhat increased compared to some baseline models, it still fulfills the requirements for real-time processing. The proposed PJ-YOLO showcases significant practical value for scenarios demanding immediate responsiveness, effectively balancing performance with speed, and holds potential for real-world applications in infrared ship detection tasks.

4.7. Performance Evaluation in Real-World Scenes

To evaluate the generalization capability of the proposed PJ-YOLO, the model trained on the SFISD dataset was utilized for detecting ships on real-world infrared images collected from the Internet. The visualization of prediction results is presented in Figure 13. In Figure 13a, both YOLOv8 and YOLOv10 fail to detect the ship target, and YOLOv9 mistakenly identifies the watermark as a ship target. In Figure 13b, YOLOv8, YOLOv9, and YOLOv10 all mistakenly identify the island on the left side of the image as a ship target. In Figure 13c, YOLOv8, YOLOv9, and YOLOv10 all exhibit numerous false detections. In comparison, the proposed PJ-YOLO successfully detects ship targets without any false alarms, demonstrating better detection capability and highlighting its enhanced generalization ability.

5. Conclusions

In this paper, we proposed a PJ-YOLO for infrared ship detection, which is dedicated to improving detection performance. In the feature extraction stage, a joint feature extraction module was designed to enhance the extraction of effective features from multiple dimensions. To enhance the model’s focus on targets of different scales, we introduced a residual deformable attention mechanism at the high level of the backbone network, improving the capture of target features. Additionally, we designed a prior auxiliary loss using prior knowledge of infrared ship brightness distribution to more accurately locate targets, enhancing the model’s robustness and improving target localization accuracy. Extensive experiments on the SFISD dataset and the InfiRray Ships dataset validate that PJ-YOLO significantly improves the detection accuracy of infrared ships.

Future research will focus on lightweighting the network structure to enhance FPS while maintaining high detection accuracy, thereby improving operational efficiency and reducing computational resource consumption. To achieve this, we will explore techniques such as network pruning, model lightweighting, operator optimization, and knowledge distillation. Additionally, to enhance the model’s generalization capability, we will also focus on collecting a more diverse set of infrared ship images, aiming to ensure that it maintains stable and efficient performance across various real-world conditions.