RAIH-Det: An End-to-End Rotated Aircraft and Aircraft Head Detector Based on ConvNeXt and Cyclical Focal Loss in Optical Remote Sensing Images

Abstract

In airport ground-traffic surveillance systems, the detection of an aircraft and its head (AIH) is an important task in aircraft trajectory judgment. However, accurately detecting an AIH in high-resolution optical remote sensing images is a challenging task due to the difficulty in effectively modeling the features of aircraft objects, such as changes in appearance, large-scale differences, complex compositions, and cluttered background. In this paper, we propose an end-to-end rotated aircraft and aircraft head detector (RAIH-Det) based on ConvNeXt-T (Tiny) and cyclical local loss. Firstly, a new U-shaped network based on ConvNeXt-T with the same performance as the Local Vision Transformer (e.g., Swin Transformer) is presented to assess the relationships among aircraft in the spatial domain. Then, in order to enhance the sharing of more mutual information, the extended BBAVectors with six vectors captures the oriented bounding box (OBB) of the aircraft in any direction, which can assist in head keypoint detection by exploiting the relationship between the local and overall structural information of aircraft. Simultaneously, variant cyclical focal loss is adopted to regress the heatmap location of keypoints on the aircraft head to focus on more reliable samples. Furthermore, to perform a study on AIH detection and simplify aircraft head detection, the OBBs of the “plane” category in the DOTA-v1.5 dataset and the corresponding head keypoints annotated by our volunteers were integrated into a new dataset called DOTA-Plane. Compared with other state-of-the-art rotated object and keypoint detectors, RAIH-Det, as evaluated on DOTA-Plane, offered superior performance.

1. Introduction

In recent years, the detection of aircraft in optical remote sensing images has become a significant task in various civilian and military applications [1,2,3]. The distribution of an airport is shown in Figure 1. Accurately predicting the orientation (or the head) of the aircraft, besides detecting the position of the aircraft, is an essential step in aircraft trajectory judgment in the airport ground-traffic monitoring system [4]. However, aircraft detection and aircraft head detection remain independent tasks. In the case of aircraft head detection in an image, first, the aircraft must be accurately detected; then, its head can be detected. These complicated steps increase the requirements of training samples to train deep learning models and reduce computational efficiency. Therefore, we consider merging aircraft detection and head detection into a unified network for end-to-end training in this paper.

With the rapid development of remote sensing systems, various high-resolution (HR) optical remote sensing images (e.g., Worldview, Quickbird, GeoEye, and Chinese GF 1–14) are being acquired daily to obtain rich and detailed structures of the Earth’s surface [5]. So far, these images have been annotated for object detection by professional teams such as UCAS-AOD [6], RSOD-Dataset [7], DOTA [8], and NWPU VHR-10 [9]. DOTA provides two types of annotations: horizontal bounding boxes (HBBs) and oriented bounding boxes (OBBs). Compared with the former, the width and height of the latter directly reflect the relative size of the target. Recently, deep learning-based algorithms have emerged as an effective strategy for object detection research, such as the well-known YOLO series [10,11,12], single-shot multibox detector (SSD) [13], RetinaNet [14], CornerNet  [15], CenterNet [16], IR R-CNN [17], rotation-sensitive regression for oriented scene text detection (RRD) [18], ROI Transformer [19], box boundary-aware vectors (BBAVectors) [20], and multiknowledge learning module (MKLM) [21], which have significantly improved detection accuracy. In nature scenes, most existing target head detection methods also directly treat head detection as a specific form of object detection. However, there is very little theory and research on target head detection in optical remote sensing images, with only a few studies on different scene tasks [4,22,23]. For example, to perform aircraft head detection, Jia et al. [4] clipped the recognized aircraft and pre-trained a head prediction module using a classification network. FGSD provides a new annotation for ship head detection [22]. Inspired by this, we annotated a new dataset for aircraft head detection.

The detection of an aircraft and its head (AIH) is a complex multitasking problem in the computer vision community, involving three main tasks: localization, classification, and head keypoint detection. Just as a human discovers an aircraft, an object detector must show where the aircraft is located, whether it is an aircraft, and where its head is pointing. This is simple for humans, but incredibly troubling for neural networks. Currently, the following several obstacles limit the ability of AIH detection tasks in HR remote sensing images: (i) These images, acquired with long-distance and wide-angle remote sensing systems, contain aircraft with significant differences. (ii) Differently from object detection in natural scenes, targets in an optical remote sensing image exist in different orientations due to the special top-down imaging perspective [24]. (iii) Many objects of interest in large-scale remote sensing images can be inevitably surrounded by complex compositions and cluttered backgrounds, which can seriously interfere with detecting the object of interest. (iv) Aircraft head detection is a challenging because the aircraft head occupies only a few pixels in a remote sensing image. Therefore, effective feature representation is essential to making AIH detection more accurate.

Since object detection is usually performed on feature space, effective feature representation is of great significance to constructing a high-performance object detection system. For this reason, various feature representation methods have been built to effectively distinguish among different types of interest objects in complex and irrelevant scenes. In general, region proposal network (RPN) [25] and feature pyramid network (FPN) [26] are incorporated to extract multiscale strong and weak semantic features. However, the dense candidate boxes generated with these methods are most often only a few boxes containing objects. As a result, background samples dominate the training process and degrade the model. In CornerNet [15] and CenterNet [16], fewer bounding boxes are obtained by estimating a keypoint-based detection pipeline, rather than defining dense candidate boxes. Rotation-equivariant detector (ReDet) [27] incorporates rotation-equivariant features, which can obtain the orientation and result with a huge reduction in model size. A recent method, box boundary-aware vectors (BBAVectors) [20], extends the keypoint-based CenterNet to rotated object detection and can capture the four vectors with mutual information of OBBs in the Cartesian coordinate system. Objects in an image are usually not independent; instead, they emerge in some available spatial relationship. Therefore, IR R-CNN [17] and HK R-CNN [28] establish intrinsic semantic and spatial location relationships among objects. More recently, Vision Transformer (ViT), based on multi-head self-attention mechanisms, can also effectively model the interaction among various spatial locations. ConvNeXt [29] can achieve the same level of scalability as Swin Transformer. Inspired by previous studies, we aim to achieve an end-to-end AIH detector by availing ourselves of the respective advantages of ConvNeXt, BBAVectors, and deep supervision techniques.

In this paper, we present an end-to-end trainable rotated aircraft and aircraft head detector (RAIH-Det) based on ConvNeXt-T and the variant cyclical focal loss (CFL). To facilitate AIH detection and aircraft head detection, we integrated the OBBs from the “plane” category of the DOTA-v1.5 dataset with our annotated aircraft head keypoints into a new dataset called DOTA-Plane. Then, we performed an evaluation, an ablation study and a comparison on the DOTA-Plane dataset. The experiments showed that the proposed RAIH-Det obtained superior performance on the DOTA-Plane dataset. Moreover, to show the universality of the proposed RAIH-Det, we detected arbitrarily orientated ships on the widely used HRSC2016 dataset. The main contributions of this work are four-fold:

1.

We propose the novel RAIH-Det, which integrates two independent tasks into a unified network: aircraft detection and aircraft head detection. RAIH-Det fully uses the spatial and context relationships to identify aircraft of various scales, orientations, shapes, and spatial distributions.

2.

We incorporated ConvNeXt-T with the same performance as the Local Vision Transformer (Swin Transformer) into a new U-shaped architecture to obtain an output feature map more distinguishable in the spatial domain. Instead of BBAVectors with the original four vectors, six vectors containing the head keypoint mutual information were adopted to generate the OBBs.

3.

We used a variant CFL to focus on more reliable training samples, thus overcoming the imbalance issue caused by the fact that the aircraft head occupies a small number of pixels in optical remote sensing images. Meanwhile, the added head keypoint mutual information can further assist in head keypoint detection by exploring the relationship between head keypoints and the OBBs.

4.

We present a new DOTA-Plane dataset consisting of OBBs of the “plane” category and aircraft head keypoints that we annotated in the DOTA-v1.5 dataset.

The remainder of this paper is organized as follows: Section 2 provides a brief review of related work. Section 3 provides an overview of the proposed method, followed by a detailed description of the framework. Section 4 describes the AIH detection dataset and the information contained therein. Section 5 outlines the comprehensive experimental results. Finally, the presentation of the proposed RAIH-Det is concluded in Section 6.

2. Related Works

2.1. Vision Backbones

Convolutional neural networks (CNNs) have become dominant in computer vision modeling. Starting with AlexNet [30] and its revolutionary performance in ImageNet image classification tasks, CNN architectures (e.g., VGGNet [31], ResNet [32], EfficientNet [33], and ConvNeXt [29]) have evolved to become increasingly powerful with greater scale, wider connections, and more complicated forms of convolution [34]. The progress of network architectures has caused performance improvements that have greatly promoted the whole field, since CNNs serve as the backbone network in various vision tasks. Recently, ViT pioneering work directly adopted transformers on a sequence of non-overlapping fixed-size image patches for image classification tasks. Compared with CNNs, ViT successfully achieves reconciliation between running speed and detection accuracy in image classification. However, vanilla ViT encountered some difficulties when employed in common computer vision tasks such as change detection, object detection, scene understanding, and semantic segmentation. Therefore, to expand the applicability of transformers as generic vision backbones [35], Swin Transformer [35] was reintroduced by using the shifted window-based block with an independent self-attention mechanism separately for each partition. The Neighborhood Attention Transformer (NAT) adopts a simple and flexible multi-head attention mechanism to localize the receptive field of each query to its nearest neighboring pixels. More recently, ConvNeXt [29] was proposed as a new CNN backbone influenced by Swin Transformer. However, it was not attention-based and managed to outperform Swin Transformer with respect to several benchmarks.

2.2. Rotated Object and Keypoint Detectors

In natural scene images, HBBs are generally adopted to describe the location of objects. However, all objects of interest in optical remote sensing images are located with different orientations due to the unique perspective of special top-down photography. Therefore, various superior rotated object detectors have emerged to more accurately capture targets using OBBs (see

Table 1. The summary table of rotated object and keypoint detectors.

Model	Is Anchor Box Set?	Number of Anchor Boxes	Keypoint Detector	Balance Problem	Computational Cost
R2CNN [36]	✓	Many	-	✓	High
RRPN [37]	✓	Many	-	✓	High
R${}^2$PN [38]	✓	Many	-	✓	High
ICN [39]	✓	Many	-	✓	High
CFC-Net [40]	✓	Only one	-	-	Moderate
CornerNet [15]	-	Free	✓	✓	Moderate
CenterNet [16]	-	Free	✓	✓	Moderate
CentripetalNet [41]	-	Free	✓	✓	Low
BBAVectors [20]	-	Free	✓	✓	Low

). Current rotated object detectors are generally extensions of horizontal object detectors. For example, the rotational region CNN (R2CNN) based on region proposal network (RPN) and ROI-pooled features [36] was proposed to produce rotational and horizontal bounding boxes. Simultaneously, rotation region proposal networks (RRPNs) [37] and rotated region proposal networks (R${}^2$PNs) [38] have been designed to obtain inclined proposals with orientation angle information. ROI Transformer [19] adopts spatial transformations in ROIs to maintain rotation invariance. ICN [39] incorporates image cascades and feature pyramid networks with multi-size convolution kernels to extract multiscale strong and weak semantic feature maps. As a result, satisfactory performance was obtained on the DOTA dataset. All these approaches involve generating a lot of anchor boxes. Meanwhile, the position offsets between the target bounding boxes and the anchor boxes are regressed to refine the position of the prediction box. Here, the anchor-based strategy is affected by the imbalance problem between negative and positive anchor boxes, which leads to slow training speed and suboptimal detection performance. The critical feature capturing network (CFC-Net) [40] is a one-stage detector with only one anchor corresponding to each position of the feature map, which is equivalent to anchor-free methods with higher inference speed.

Compared with traditional keypoint detectors [42,43,44], deep learning methods are more general and can be categorized into the following: (i) the keypoint position is directly regressed, and (ii) heatmap regression followed by post-processing is adopted to search for the keypoint with maximal heat value, which is also adopted in this study. Regarding approaches based on heatmap regression, Tompson et al. [45] proposed a multi-resolution framework that generates heatmaps representing the pixel-by-pixel likelihood of keypoints. CornerNet [15] uses heatmaps to capture the top-left and bottom-right corner points of HBBs. Corner points are grouped by comparing the embedding distance of the points for each object. Duan’s CenterNet [46] detects a pair of corners and the center point of a bounding box. ExtremeNet [47] was presented as able to locate the extreme and center points of the boxes. Both approaches adopt the center information and geometry of box points to group and match points. However, these works relied on post-grouping processing, which is a computationally expensive and time-consuming operation. Without the post-grouping process, Zhou’s CenterNet [16] directly regresses the bounding box parameters at the center point. To further improve the performance of keypoint-based detectors, CentripetalNet [41] adopts centripetal shift to pair corner keypoints from the same target. A recent method, BBAVectors [20], extends the keypoint-based CenterNet to rotated object detection in a single stage without anchor boxes. BBAVectors specializes in improving the performance of center keypoint detection and OBB regression. Our AH detection is a complex multitasking problem consisting of three main tasks: localization, classification, and head keypoint detection. Therefore, we extended BBAVectors to aircraft and aircraft head detection tasks.

3. Methods

The overall architecture of the proposed detector (RAIH-Det) is shown in Figure 2. Our model begins with a U-shaped architecture to obtain the feature map for each input image and then feeds them to the detection module to predict the parameters of the OBB and the head keypoint of the aircraft. Finally, the formula of the loss function is defined.

The proposed RAIH-Det has three main components: (1) a U-shaped architecture based on ConvNeXt-T, which merges the multistage features with the interaction among different spatial locations; (2) a box parameter prediction layer with six vectors (extended BBAVectors), which groups the head keypoint mutual information into semantic features (visual attributes) to generate the OBB; (3) a head keypoint detection layer, which applies a likelihood heatmap to obtain correspondence for the location of each keypoint and utilizes variant CFL to focus on more reliable training samples. The inference detail of the proposed RAIH-Det for aircraft and aircraft head detection is summarized in Algorithm 1.

Algorithm 1: RAIH-Det: An End-to-End Rotated Aircraft and Aircraft Head Detector Based on ConvNeXt and Cyclical Focal Loss in Optical Remote Sensing Images.

Input: The input images $I_{input}\in {\Re }^{H\times W\times 3}$.  Output: Aircraft head keypoints and rotated bounding box with score. 1 ⊳  U−shaped architecture with ConvNeXt−T. 2   (i) The contracting path (left side): 3        Output basic blocks 1−5 in ConvNeXt−T; 4        The feature maps $f_l\in {\Re }^{H/s_l\times W/s_l\times C_l},(l=2,3,4,5)$. 5   (ii) The contracting path (right side): 6        Obtain the feature map $M\in {\Re }^{H/s_2\times W/s_2\times 96}$. 7 ⊳  Detection Module. 8   Aircraft Head keypoint prediction: 9        Head keypoint heatmaps $\tilde{\mathbf{H}}\in {\Re }^{H/s_2\times W/s_2\times 2}$; Compute $L_{box}$ to learn the heatmap of head keypoints by Equations (4)–(6). 10      Offsets ${\mathbf{O}}_h\in {\Re }^{H/s_2\times W/s_2\times 2}$; Compute the difference $L_{off}^h$ between quantified floating head keypoints and integer head keypoint by Equation (9). 11   Aircraft box prediction: 12       Box parameters $\tilde{\mathbf{B}}\in {\Re }^{H/s_2\times W/s_2\times 14}$; Compute $L_{box}$ to optimize the box parameters with six vectors at the center point by Equation (3). 13       Heatmaps of center keypoints $\tilde{\mathbf{C}}\in {\Re }^{H/s_2\times W/s_2\times 2}$; Compute $L_{cen}$ to learn the heatmap of center keypoints. 14       Offsets ${\mathbf{O}}_c\in {\Re }^{H/s_2\times W/s_2\times 2}$; Compute the difference $L_{off}^c$ between quantified floating center keypoints and integer center keypoint. 15       The orientation maps $\mathbf{\vartheta }$$\in {\Re }^{H/s_2\times W/s_2\times 1}$; Compute the difference $L_{\vartheta }$ between predicted and ground-truth orientation classes by Equation (11)

3.1. U-Shaped Architecture Based on ConvNeXt-T

When the complexity of scene visibility is formed by multiple aspects, including multiscale shape, quantity, and position of aircraft and other targets in an airport, AIH detection can exploit the global and local relationships among the targets to enhance the detection performance [48]. The success of ViT (or Swin Transformer) suggests that multi-head self-attention mechanisms can model the interaction among different spatial locations. The recently developed ConvNeXt [29] can achieve the same performance level (as Swin Transformer) and improve the running speed. In response to such observations, we built a new U-shaped architecture to learn multiscale features for detecting objects with specific scales.

The proposed U-shaped architecture is also composed of a contracting path (left side) and an expansive path (right side). The contracting path adopts a new pre-trained ConvNeXt-T (Tiny) architecture consisting of a repeating block of $7\times 7$ depthwise convolutions (a “patchify” layer similar to Swin Transformer), one BatchNorm2d, two $1\times 1$ convolutions, and a Gaussian error linear unit (GELU) layer between the two $1\times 1$ convolutions. Aircraft at different airports present heterogeneity in shape and size. Therefore, the outputs of ConvNeXt-T basic blocks 1–5 are applied as a backbone. Given an image $I_{input}\in {\Re }^{H\times W\times 3}$ acquired with a remote sensing system, where H, W, and 3 denote the height, width, and channel number of the image, respectively, by down-sampling the ratio $s_l=2^l$, corresponding to the input image at different layers $l(l=2,3,4,5)$, the feature maps $f_l\in {\Re }^{H/s_l\times W/s_l\times C_l}$ with depths $C_l$ 96, 192, 384, and 768 are sequentially obtained. To halve the number of feature channels, each step of the expansive path adopts an “add” block that consists of linear up-sampling and $1\times 1$ convolution layers. The bilinear interpolation method is applied in the up-sampling layer. Additionally, each layer is followed by one InstanceNorm2d and a GELU layer and is concatenated with a feature map cropped from the corresponding contracting path. Finally, the output feature map $M\in {\Re }^{H/s_2\times W/s_2\times C_2}(C_2=96)$ can be obtained.

3.2. Detection Module

As described in Figure 2, the output feature map M is sent to the detection module, which consists of two submodules, aircraft box prediction and head keypoint prediction:

• Aircraft box prediction: box parameters $\tilde{\mathbf{B}}\in {\Re }^{H/s_2\times W/s_2\times 14}$, heatmaps of center keypoints $\tilde{\mathbf{C}}\in {\Re }^{H/s_2\times W/s_2\times 2}$, offsets ${\mathbf{O}}_c\in {\Re }^{H/s_2\times W/s_2\times 2}$, and orientation classes $\mathbf{\vartheta }$$\in {\Re }^{H/s_2\times W/s_2\times 1}$ (see Section 3.3).
• Head keypoint prediction: head keypoint heatmaps $\tilde{\mathbf{H}}\in {\Re }^{H/s_2\times W/s_2\times 2}$ and offsets ${\mathbf{O}}_h\in {\Re }^{H/s_2\times W/s_2\times 2}$.

Six prediction branches are computed by the feed-forward network (FFN) (i.e., n-layer perceptron with GELU activation function, hidden dimension, and linear projection layer).

(1) Box parameters with six vectors: Existing rotated object detectors often regress five parameters $Center+wh+\theta$ (coordinates of the center point, width, height, and rotated angle) and utilize smooth L1 loss as the loss function to represent the OBB of an object in any direction. However, such an approach has three fundamental problems in practice: (1) The loss discontinuities are induced by variations in angle. When the angle is close to its range limits, the loss value skips. The position of the reference rectangle slightly changes, but its angle significantly changes due to parameterization. (2) For each target, the width w and height h of its OBB change in the five-parameter rotating coordinate system and make the case more difficult. (3) Most approaches employ ln-norm loss to independently optimize the five parameters, but the five variables are actually correlated.

Hence, Wen et al. [49] introduced a rotation-sensitive detector (RSDet) based on an eight-parameter box. RSDet modulates the rotational loss to eliminate the inconsistent parameter regression and the loss discontinuity. A recent method, BBAVectors [20], can also represent the OBBs of objects in any direction in optical remote sensing images and adopts the regression model of four vectors with mutual information. Specifically, Cartesian coordinates are first established, with the center point $\mathbf{c}=(x_c,y_c)$ of the object as the origin point. Afterwards, four vectors (top $\mathbf{t}=(x_t,y_t)$, right $\mathbf{r}=(x_r,y_r)$, bottom $\mathbf{b}=(x_b,y_b)$, and left $\mathbf{l}=(x_l,y_l)$) from the coordinate origin to the four different side midpoints of the OBB can be represented (see Figure 3a). With these vectors, the top-to-left corner ($\overline{\mathbf{tl}}$), top-to-right corner ($\overline{\mathbf{tr}}$), bottom-to-right corner ($\overline{\mathbf{br}}$), and bottom-to-left corner ($\overline{\mathbf{bl}}$) of the OBB in the image can be calculated as follows:

$$\begin{array}{cc}\hfill \overline{\mathbf{tl}}& =(\mathbf{t}+\mathbf{l})+\mathbf{c},\overline{\mathbf{tr}}=(\mathbf{t}+\mathbf{r})+\mathbf{c},\hfill \\ \hfill \overline{\mathbf{br}}& =(\mathbf{b}+\mathbf{r})+\mathbf{c},\overline{\mathbf{bl}}=(\mathbf{b}+\mathbf{l})+\mathbf{c}.\hfill \end{array}$$

(1)

Moreover, when the long axis of the OBB is close to the origin of the coordinate system, the OBB of the arbitrarily oriented target can be assumed as the HBB. The top-to-left corner ($\overline{\mathbf{tl}}$), top-to-right corner ($\overline{\mathbf{tr}}$), bottom-to-left corner ($\overline{\mathbf{bl}}$), and bottom-to-right corner ($\overline{\mathbf{br}}$) of the HBB in the image can be written as

$$\begin{array}{cc}\hfill \overline{\mathbf{tl}}& =(x_c-w_{hbb}/2,y_c-h_{hbb}/2),\overline{\mathbf{tr}}=(x_c+w_{hbb}/2,y_c-h_{hbb}/2),\hfill \\ \hfill \overline{\mathbf{br}}& =(x_c+w_{hbb}/2,y_c+h_{hbb}/2),\overline{\mathbf{bl}}=(x_c-w_{hbb}/2,y_c+h_{hbb}/2),\hfill \end{array}$$

(2)

where $w_{hbb}$ and $h_{hbb}$ represent the width and height of the HBB surrounding the arbitrarily oriented targets, respectively.

Except for the overall structural information of the aircraft, local structural (or mutual) information belonging to the aircraft head can also provide semantic features (visual attributes) in a concise and efficient manner. Especially when the texture and structure of a single object are not obvious, the information is instrumental in human–computer interaction and knowledge transfer. To share more mutual information, we intentionally designed six vectors based on BBAVectors $\tilde{\mathbf{B}}$ ($\mathbf{t}$, $\mathbf{b}$, $\mathbf{r}$, $\mathbf{l}$, ${\mathbf{h}}_{\mathbf{1}}$, and ${\mathbf{h}}_{\mathbf{2}}$) instead of four vectors (see Figure 3b), where ${\mathbf{h}}_{\mathbf{1}}$ represents the vector from the aircraft head keypoint to the ordinate origin (center point) and ${\mathbf{h}}_{\mathbf{2}}$ represents the vector from the nearest midpoint of the head keypoint to the head keypoint. The added head mutual information can assist in the detection of the head keypoint by exploiting the relationship between the head keypoint and the OBB. Smooth L1 loss was also adopted to optimize the box parameters with six vectors in the center point:

$$L_{box}({\mathbf{b}}^p,{\mathbf{b}}^{gt})=\frac{1}{N}\sum _i^{N}Smoot{h}_{L_1}({\mathbf{b}}_i^p-{\mathbf{b}}_i^{gt}),$$

(3)

where ${\mathbf{b}}^p$ and ${\mathbf{b}}^{gt}$ refer to six vectors of the predicted and ground-truth bounding boxes, respectively.

(2) Head keypoint detection: Most of the top-performing methods for keypoint detection generate likelihood heatmaps that correspond to the location of each keypoint (e.g., facial landmarks [50,51]) in the input image. Inspired by these, we adopted heatmaps to detect the head keypoint of an airplane in optical remote sensing images. Figure 4 shows the heatmap annotation of an aircraft and its head keypoint.

Ground-truth of head keypoint. Let us suppose that $\tilde{\mathbf{h}}=(x_h,y_h)$ is an aircraft head keypoint. In this case, we place a 2D Gaussian blob $\exp (-\frac{{(x_p-x_h)}^2+{(y_p-y_h)}^2}{2{\sigma }^2})$ centered in each keypoint $\tilde{\mathbf{h}}$ and construct a ground-truth heatmap $\tilde{\mathbf{P}}\in {\Re }^{K\times \frac{H}{s_2}\times \frac{W}{s_2}\times 3}$. $\tilde{p}=(x_p,y_p)$ is the index of pixel points on $\tilde{\mathbf{P}}$. ${\sigma }^2$ usually depends on the box size-adaptive standard deviation [16]. Considering that the aircraft head occupies only a small area in the entire fuselage and that the position of the head keypoint may be assigned to multiple objects due to the overlapping boxes, we can select a box $\gamma$ with the minimal area as the corresponding regression aircraft head. More specifically, we first calculate the shortest distance (i.e., the radius of the head area) from the head keypoint to the four sides of the OBB and then construct a square area with the shortest distance.

Heatmap regression of head keypoint. When training the heatmaps of the head, only the keypoint $\tilde{\mathbf{h}}$ is positive, and additional bumps such as the sample points are negative. Directly learning positive head keypoints is difficult due to imbalance issues. Focal loss has proven to be effective in tasks with imbalanced class data. However, it cannot be fully guaranteed that all training is optimal, because different datasets have different degrees of balance [52]. In particular, the radius of the head area is quite small and not fixed, which provides a more variable degree of balance to calculate the extent of the heatmap. Recently, CFL [52] based on the principle of general cyclical training has been proposed for balanced, imbalanced, and long-tailed datasets. Following the work by [52], we reduce the penalties for points inside Gaussian bumps and learn the heatmap of head keypoints using variant CFL.

$$\begin{array}{c}\hfill L_{head}=-\frac{1}{N^{*}}\left[\xi \sum _t{L}_{hc}\left(p_t\right)+(1-\xi )\sum _t\ell {(1-{\widehat{p}}_t)}^{\alpha }\log \left({\widehat{p}}_t\right)\right],\end{array}$$

(4)

$$L_{hc}\left(p_t\right)=-{(1+p_t)}^{{\gamma }_{hc}}\log \left(p_t\right),$$

(5)

$$\begin{array}{c}\hfill {\widehat{p}}_t=\left\{\begin{array}{cc}{p}_t,\hfill & if\hspace{0.17em}{\tilde{p}}_t=1\hfill \\ 1-p_t,\hfill & otherwise,\hfill \end{array}\right.\end{array}$$

(6)

$$\begin{array}{c}\hfill \ell =\left\{\begin{array}{cc}1,\hfill & if\hspace{0.17em}{\tilde{p}}_t=1\hfill \\ {(1-{\tilde{p}}_t)}^{\beta },\hfill & otherwise,\hfill \end{array}\right.\end{array}$$

(7)

where $N^{*}$ is the total number of aircraft heads; t indexes pixel positions on the feature map; $L_{hc}$ causes the loss to focus on more reliable training samples; p and $\tilde{p}$ represent the predicted and ground-truth heatmaps, respectively; the hyperparameters $\alpha$ and $\beta$ quantify the contribution of each point. Empirically, we select $\alpha =2$ and $\beta =4$ as in Ref. [9]. The parameter $\xi$ varies with the training epoch as follows:

$$\begin{array}{c}\hfill \xi =\left\{\begin{array}{cc}1-f_c\frac{e_i}{e_t},\hfill & if\hspace{0.17em}\frac{e_i}{e_t}\le \frac{1}{f_c}\hfill \\ (f_c\frac{e_i}{e_t}-1)/(f_c-1),\hfill & otherwise,\hfill \end{array}\right.\end{array}$$

(8)

where $f_c$ denotes the cyclical factor that offers variability to the cyclical schedule ($f_c=4$), and $e_i$ and $e_t$ represent the current training epoch number and the total number of training epochs, respectively.

Head keypoint offset. During the implementation of the algorithm, we are required to extract the peak points of the predicted heatmap $\tilde{\mathbf{h}}$ as the head keypoint positions of the aircraft. This head keypoint $\tilde{\mathbf{h}}$ is an integer. However, the down-scaling of points from the input image to the output heatmap is no longer an integer but a floating-point number. Therefore, an offset map ${\mathbf{O}}_h\in {\Re }^{K\times \frac{H}{s_2}\times \frac{W}{s_2}}$ is calculated for the difference between quantified floating head keypoints and integer head keypoint, i.e., $\mathbf{o}=(\frac{x_h}{s_2}-⌞\frac{x_h}{s_2}⌟,\frac{y_h}{s}-⌞\frac{y_h}{s_2}⌟)$. The offset is optimized with smooth $L_1$ loss.

$$L_{off}^h(\mathbf{o},{\mathbf{o}}^{*})=\frac{1}{N^{*}}\sum _i^{N}Smoot{h}_{L_1}(o_i-o_i^{*}),$$

(9)

where $o^{*}$ is the ground-truth offset. As shown in Figure 3b, it is intuitive and reasonable that a pair of vectors $h_1$ and $h_2$ belonging to the fuselage orientation share the keypoint of that aircraft head. Since we can decode the corresponding center (starting point) of a predicted vector from its location and the relationship between it and other vectors with extended BBAVectors, it is easy to compare whether the starting points of a pair of vectors $h_1$ and $h_2$ are close enough and close to the keypoint of aircraft head. To match the vectors $h_1$ and $h_2$, we further adjust the head keypoint position of the aircraft using their locations and the relationship between vectors with extended BBAVectors.

The detection of aircraft center keypoints is governed by $L_{cen}$ and $L_{off}^c$, as is the detection of head keypoints.

3.3. Loss Function

To aid the learning of the proposed RAIH-Det, the multitasking loss function can be divided into six parts as follows:

$$\begin{array}{c}\hfill L_{total}=L_{box}+L_{cen}+L_{off}^c+L_{\vartheta }+L_{head}+L_{off}^h,\end{array}$$

(10)

where $L_{box}$ is computed to learn the box parameters with six vectors using Equation (3); $L_{head}$ and $L_{cen}$ are computed to regress the keypoint heatmap using Equations (3)–(6), respectively; $L_{off}^c$ and $L_{off}^h$ are calculated for the difference between quantified floating and integer head keypoints using Equation (9), respectively; the orientation class $L_{\vartheta }$ is derived from the binary cross-entropy, which can help the network to capture precise OBBs when encountering corner situations [20].

$$L_{\vartheta }=-\frac{1}{N}\sum _k(\tilde{\vartheta }\log \left(\vartheta \right)+(1-\tilde{\vartheta })\log (1-\vartheta )),$$

(11)

$$\begin{array}{c}\hfill \tilde{\vartheta }=\left\{\begin{array}{cc}0\left(HBB\right),\hfill & if{f}_{IOU}(OBB,HBB)>0.95\hfill \\ 1,\hfill & otherwise,\hfill \end{array}\right.\end{array}$$

(12)

where $\vartheta$ and $\tilde{\vartheta }$ refer to predicted and ground-truth orientation classes, respectively; $f_{IOU}$ denotes the intersection over union between OBB and HBB; and $f_{IOU}(OBB,HBB)>0.95$ indicates that the long axis of the OBB is close to the origin of the coordinate system and that the arbitrarily oriented target can be equivalent to the HBB.

4. Datasets

DOTA [8] is a large-scale dataset for object detection in optical remote sensing images, with images collected at multiple resolution levels. DOTA images range in resolution from $800\times 800$ pixels to $4000\times 4000$ pixels. The dataset provides two types of annotations: HBBs and OBBs. Compared with the former, the width and height of the latter directly reflect the relative size of the aircraft. DOTA-v1.5 comprises 2806 images of 16 target categories from the Google Earth, GF-2, and JL-1 satellites provided by the China Center for Resources Satellite Data and Application. This is an updated version of DOTA-1.0, where tiny instances (smaller than 10 pixels) are additionally annotated. The OBBs in DOTA are more flexible and easier to understand and are annotated as $(\{(x_i,y_i),i=1,2,3,4\},c,d)$, where $(x_i,y_i)$ are the positions of the OBB vertices in the optical remote sensing image, c denotes the target category, and d is the difficulty level of target discrimination. Following the work by [53], we selected the “plane” category with 269 images from DOTA-v1.5 as our dataset for the AIH detection study, named DOTA-Plane. It was divided into a training set (198 images) and a test set (71 images). Each aircraft instance contained two labels: the OBB label and the head keypoint label. The former was derived from DOTA-v1.5, and the latter was annotated by our volunteers.

Since the head is only a small part of the entire fuselage, our volunteers spent two weeks manually marking the ground truth of the head center keypoints of all types of aircraft in DOTA-Plane with LabelMe software (https://github.com/wkentaro/labelme, accessed on 9 May 2016) to facilitate aircraft head detection. During the marking process, the annotations sometimes resulted to be coarse or erroneous, because the volunteers occasionally missed an aircraft or were inaccurate in marking the head center location. Therefore, to avoid such errors, three other volunteers visually cross-checked all annotations. The coordinates of the OBBs derived from DOTA-v1.5 were used as a reference for further correction of the head center position. As shown in Figure 5, given a plane with an OBB and the head center position $(\{(x_i,y_i),i=1,2,3,4\},x_h,y_h,c,d)$, $(x_h,y_h)$ denotes the head center point of that plane. The center position of the plane, $(x_c,y_c)$, is calculated using the OBB. The orientation of the plane can be expressed with the vector $h_1=(x_c-x_h,y_c-y_h)$, pointing from the center position of the plane to the head center position of the plane.

5. Experiments and Results

This section first presents the implementation details (parameter settings) of the proposed RAIH-Det and the evaluation metrics used in the experiments. A series of comparative experiments were conducted to verify the effectiveness of the proposed RAIH-Det on the DOTA-Plane dataset. The results obtained in AIH detection are then discussed.

5.1. Implementation Details

In our experiments, owing to the limitation issues of the GPU memory, the original images were cropped into small patches of size $600\times 600$ and stride 300. Most of the patches, including the aircraft, corresponding to a small fraction of the large HR optical remote sensing images in the training set, were occupied by empty backgrounds. Therefore, map shifting and spinning were performed while sampling some of the aircraft samples to significantly increase the number of data and ensure the diversity of the dataset. All input images were resized to $608\times 608$ pixels and used in the training and testing stages.

We implemented RAIH-Det using the deep learning framework PyTorch library. Adam optimizer was employed for training. The learning rate was set to 1.25 × 10${}^{-4}$ until 100 epochs were trained. The batch size of RAIH-Det was set to 4, and ConvNeXt-T’s pre-trained weight parameters were loaded. All experiments were performed on a single workstation equipped with a 4.20 GHz Intel (R) Core (TM) i7-7700K CPU with 32 GB RAM and an NVIDIA GeForce Titan Xp GPU with 12 GB VRAM.

5.2. Evaluation of AIH Detection

Precision ($Prec$) and recall ($Rec$) comprehensively reflect the detection accuracy and completion rate in the aircraft detection task, respectively. The related formulations are as follows:

$$Prec=\frac{N_{TP}}{N_{TP}+N_{FP}},$$

(13)

$$Rec=\frac{N_{TP}}{N_{TP}+N_{FN}},$$

(14)

where $N_{TP}$ denotes the number of true positives and reflects properly detected aircraft, $N_{FP}$ indicates the number of false positives and reflects false alarms, and $N_{TP}+N_{FN}$ is the number of all ground-truth values. Precision and recall values differ depending on the confidence threshold. The precision–recall curve (PRC) can then be drawn with recall values on the abscissa and precision values on the ordinate. Average precision ($AP$) is not the average value of precision but considers precision and recall rates and is the area under the PRC. In our experiments, $AP$ is a metric calculated at $IoU=0.5$. Simultaneously, F1-score is comprehensively calculated as the harmonic average of precision and recall. Furthermore, a standard evaluation metric (percentage of correct keypoints (PCK) [54]) was used to evaluate the aircraft head keypoints. Given a set of predicted and ground-truth head keypoint pairs $k={\left\{({\widehat{k}}_m^{\prime },k_m^{\prime })\right\}}_{m=1}^M$,

$$PCK\left(k\right)=\frac{1}{M}\sum _{M=1}^{m}1[\parallel {\widehat{k}}_m^{\prime }-k_m^{\prime }\parallel \le \eta ·\max (w_b,h_b)],$$

(15)

where $w_b$ and $h_b$ denote the width and height of the OBB, respectively, and $\eta$ is a tolerance factor, with $\eta \in (0.5,0.25,0.1)$.

5.3. Ablation Studies

We developed a serial of ablation experiments to evaluate the contribution of each component to the performance of RAIH-Det. All experiments followed the same data augmentation strategy. The relevant settings are discussed in the following:

• RAIH-Det (W): U-shaped architecture with ConvNeXt-T + detection module (without head keypoint prediction) + box parameters with four vectors (without head keypoint mutual information).
• RAIH-Det (WW): U-shaped architecture with ConvNeXt-T + detection module (without aircraft box prediction).
• RAIH-Det (Res-HW): U-shaped architecture with ResNet101 + detection module (with head keypoint prediction) + box parameters with four vectors (without head keypoint mutual information).
• RAIH-Det (Res-HH): U-shaped architecture with ResNet101 + detection module (with head keypoint prediction) + box parameters with six vectors (with head keypoint mutual information).
• RAIH-Det (HW): U-shaped architecture with ConvNeXt-T + detection module (with head keypoint prediction) + box parameters with four vectors (without head keypoint mutual information).
• RAIH-Det: U-shaped architecture with ConvNeXt-T + detection module (with head keypoint prediction) + box parameters with six vectors (with head keypoint mutual information).

Backbone.

Table 2. Ablation studies of RAIH-Det on DOTA-Plane dataset. Ablations were performed on aircraft box prediction (ABP), head keypoint prediction (HKP), box parameters with four vectors (without head keypoint mutual information) (BPFVs), and box parameters with six vectors (BPSVs). PCK, AP, and F1-score are reported.

Model	Modules					Head Keypoint PCKs (%)				Detection Results (%)
	ABP	HKP	BPFVs	BPSVs		$\mathit{\eta }@0.5$	$\mathit{\eta }@0.25$	$\mathit{\eta }@0.1$		AP (%)	F1 (%)
RAIH-Det (W)	✓	-	✓	-		-	-	-		92.40	88.52
RAIH-Det (WW)	-	✓	-	-		89.32	87.73	84.21		-	-
RAIH-Det (Res-HW)	✓	✓	✓	-		87.25	84.83	80.65		87.18	84.94
RAIH-Det (Res-HH)	✓	✓	-	✓		88.83	86.83	82.75		87.89	85.44
RAIH-Det (HW)	✓	✓	✓	-		88.89	86.98	83.21		88.49	85.62
RAIH-Det	✓	✓	-	✓		90.27	88.41	84.41		89.14	86.13

presents comparisons with other backbones, including both the ResNet-based and ConvNeXt-based ones, using DOTA-Plane training. RAIH-Det and RAIH-Det (HW) noticeably surpassed RAIH-Det (Res-HH) and RAIH-Det (Res-HW) with similar detection modules: RAIH-Det improved over RAIH-Det (Res-HH) using ResNet101 by +1.25% AP; +0.69% F1; and +1.38%, +1.88%, and +1.66% PCKs ($\eta @0.5$, $\eta @0.25$, and $\eta @0.1$, respectively). RAIH-Det (HW) improved over RAIH-Det (Res-HW) using ResNet101 by +1.31% AP; +0.68% F1; and +1.64%, +2.15%, and +2.56% PCKs ($\eta @0.5$, $\eta @0.25$, and $\eta @0.1$, respectively). According to these results, it can be seen that ConvNeXt-T improved the model’s ability. Figure 6 presents the localization with RAIH-Det (Res-HH) and RAIH-Det using class activation maps (CAMs) [55,56] and heatmaps. With the help of ConvNext-T, the relationship among the various spatial positions of AIH can be obtained more effectively.

Parameter settings of the detection module. Compared with RAIH-Det (HW), the AP, F1, and PCKs ($\eta @0.5$, $\eta @0.25$, and $\eta @0.1$) of RAIH-Det, which includes the U-shaped architecture with ConvNeXt-T and the box parameters with six vectors (with head keypoint mutual information), were improved by +0.65%; +0.51%; and +1.38%, +1.43 %, and +1.2%, respectively (see Table 2). According to these results, the added head mutual information can improve the performance of the proposed method.

In the detection module, the dimension of each prediction branch is a very important parameter, which directly affects the AIH detection results of each aircraft. Therefore, we discuss the parameter settings of the FFN (i.e., n-layers perceptron with GELU activation function, hidden dimension, and linear projection layer) for six prediction branches.

Table 3. The experiments on the choice of parameters n in the prediction branch.

Heatmap C	Box Param B	Orientation #	Offset ${\mathbf{O}}_{\mathbf{c}}$	Heatmap H	Offset ${\mathbf{O}}_{\mathbf{h}}$	AP (%)	F1 (%)	$\mathit{\eta }@0.5$ (%)	$\mathit{\eta }@0.25$ (%)	$\mathit{\eta }@0.1$ (%)	Average Test Time (s)
3	3	3	3	3	3	87.98	85.12	88.81	86.63	81.86	0.13
3	1	1	1	1	1	88.14	85.64	88.97	86.66	82.27	0.11
1	1	1	1	1	1	89.14	86.13	90.27	88.41	84.41	0.08

reports the AIH detection results under different parameter settings. As a comparison, when the parameters of six prediction branches were set to 1, the best result was obtained. Note that higher-dimensional FFNs also required a longer test time.

Suitable box $\gamma$. In aircraft head keypoint detection based on heatmaps and cyclical focal loss, the aircraft head occupies only a small area of the entire fuselage. The head keypoint position can be assigned to multiple objects due to the overlapping boxes. Thus, we conducted experiments to search for the suitable box $\gamma$ as the corresponding regression aircraft head. The experimental results are shown in

Table 4. Ablation studies of head keypoint detection. ${\gamma }_1$ and ${\gamma }_2$ denote the rectangular areas, and their width and height were obtained with the width and height, or width/3 and height/3 of the OBB, respectively. On the other hand, ${\gamma }_3$ represents a square area, and its width and height were obtained by calculating the shortest distance from the head keypoint to the four sides of the OBB.

Box Size	${\mathit{\gamma }}_1$	${\mathit{\gamma }}_2$	${\mathit{\gamma }}_3$
$\eta @0.5$	89.14	89.68	90.27
$\eta @0.25$	87.61	87.87	88.41
$\eta @0.1$	83.35	83.90	84.41

. When box $\gamma$ was a rectangular area ${\gamma }_1$ (width and height were obtained with the width and height of the OBB), the head keypoint PCKs were close to 89.14 % ($\eta @0.5$), 87.61% ($\eta @0.25$), and 83.35% ($\eta @0.1$). As box $\gamma$ became smaller ($\gamma ={\gamma }_2$), the head keypoint PCKs gradually increased. Compared with ${\gamma }_1$ and ${\gamma }_2$, ${\gamma }_3$ improved the head keypoint PCKs by 90.27% ($\eta @0.5$), 88.41% ($\eta @0.25$), and 84.41% ($\eta @0.1$), respectively.

Focal loss (FL) and cyclical focal loss (CFL). The ablations of FL-based and CFL-based RAIH-Det in the two tasks are reported in

Table 5. Comparison results of the proposed RAIH-Det based on focal loss (RAIH-Det (FL)) and RAIH-Det based on cyclical focal loss (RAIH-Det (CFL)).

Method	AP (%)	F1 (%)	$\mathit{\eta }@0.5$ (%)	$\mathit{\eta }@0.25$ (%)	$\mathit{\eta }@0.1$ (%)
RAIH-Det (FL)	88.14	85.01	88.32	86.40	82.30
RAIH-Det (CFL)	89.14	86.13	90.27	88.41	84.41

. On the DOTA-Plane test set, the RAIH-Det based on CFL outperformed the RAIH-Det based on CFL by +1.0% AP; +1.1% F1; and +1.95%, +2.01%, and +2.11% PCKs ($\eta @0.5$, $\eta @0.25$, and $\eta @0.1$, respectively). According to these results, it can be seen that CFL can improve the model’s ability. Note that the RAIH-Det based on FL often greatly fluctuated in the training process.

The success of RAIH-Det not only further suggests the effectiveness of the U-shaped architecture with ConvNeXt-T and the box parameters with six vectors but also proves the gain effect of their combination.

5.4. Comparison with the State-of-the-Art Models

We evaluated the performance of the proposed RAIH-Det on the DOTA-Plane dataset with comparisons to seven state-of-the-art rotated object and keypoint detectors: RetinaNet-O (OBB) [57], CenterNet [16], CentripetalNet [41], BBAVectors [20], CFC-Net [40], ReDet [27], and MKLM [21].

Table 6. Performance comparisons on DOTA-Plane test set. AHK denotes aircraft head keypoint. * denotes that the prediction module performs additional fine-tuning procedures according to the task of aircraft head keypoint detection.

Detector Task	Model	Head Keypoint PCKs (%)				Detection Results (%)
		$\mathit{\eta }@0.5$	$\mathit{\eta }@0.25$	$\mathit{\eta }@0.1$		AP	F1
Aircraft detection	RetinaNet-O	-	-	-		74.29	74.52
	CFC-Net	-	-	-		85.18	83.15
	ReDet	-	-	-		87.54	85.68
	BBAVectors	-	-	-		90.21	87.77
	MKLM	-	-	-		88.23	86.11
	RAIH-Det (W)	-	-	-		92.40	88.52
AHK detection	CenterNet *	85.56	83.47	79.98		-	-
	CentripetalNet *	88.23	86.11	82.59		-	-
	BBAVectors *	87.50	85.83	81.79		-	-
	RAIH-Det (WW)	89.32	87.73	84.21		-	-
AIH detection	RAIH-Det	90.27	88.41	84.41		89.14	86.13

reports the OBB results on the DOTA-Plane test dataset. Figure 7 shows the performance metrics of different methods.

In the aircraft detection task, RAIH-Det (W) achieved 92.40% AP and 88.52% F1, outperforming RetinaNet-O, BBAVectors, CFC-Net, ReDet, and MKLM. RetinaNet-O had a lower performance than the other methods, reaching the lowest AP and F1 of 74.29% and 74.52%, respectively. CFC-Net adopts rotation equivariance to generate oriented anchors from horizontal anchors to achieve better spatial alignment with oriented objects; it obtained 85.18% AP and 83.15% F1. MKLM employs two types of knowledge, internal and external, and interacts with information to adaptively improve Fast RCNN’s original feature representation; it achieved better performance, with 88.23% AP and 86.11% AP. ReDet can extract rotation-equivariant features to train an accurate object detector, obtaining 87.54% AP and 85.68% F1. However, aircraft detection is time-consuming. Unlike these methods, BBAVectors represents a new concept for rotated object detection in remote sensing images and is also a keypoint-based detector with box parameters with four boundary-aware vectors. BBAVectors could yield a better performance, with 90.21% AP and 87.77% F1 on the DOTA-Plane dataset. The box regression approach of BBAVectors was extended to the task of the detection of aircraft and aircraft head. As illustrated in Figure 8, the proposed RAIH-Det (W) could more effectively distinguish different types of aircraft from optical remote sensing images with complex and irrelevant background.

For the aircraft head keypoint task, we selected the same box as the corresponding regression aircraft head area for all comparison methods (CenterNet, CentripetalNet, and BBAVectors). As shown in Table 6, RAIH-Det (WW) achieved 89.32%, 87.73%, and 84.21% PCKs ($\eta @0.5$, $\eta @0.25$, and $\eta @0.1$, respectively), outperforming CenterNet, CentripetalNet, and BBAVectors. Unlike other methods, CentripetalNet introduces the corner point-matching strategy using the centripetal shifts and locations of corner points; it obtained 88.23%, 86.11%, and 82.59% PCKs ($\eta @0.5$, $\eta @0.25$, and $\eta @0.1$, respectively).

Furthermore, we achieved the consolidation of AIH detection into a unified network (i.e., RAIH-Det) for end-to-end training. As reported in Table 6, RAIH-Det achieved 89.14% AP; 86.13% F1; and 90.27%, 88.41%, and 84.41% PCKs ($\eta @0.5$, $\eta @0.25$, and $\eta @0.1$, respectively). This result shows that learning head mutual information with the extended BBAVectors can improve head keypoint detection with RAIH-Det. In reference [58], the authors indicate that loss addition inevitably leads to better results in some tasks due to the complexity of different tasks, but to greater errors in other tasks. As shown in Table 6, the detection results of aircraft with RAIH-Det were lower detection results than those obtained in a single task (RAIH-Det (W)), but RAIH-Det also provided better performance than other state-of-the-art object detection methods. Figure 9 and Figure 10 show part of the visualization results of RAIH-Det.

Extensive experiments on different detectors and representation methods demonstrate that the proposed RAIH-Det can effectively optimize the representation learning method for AIH detection and achieve consistent performance improvements. Next, we will discuss whether the proposed method can be applied to detecting other arbitrarily orientated objects (e.g., ships). The effectiveness of the proposed rotated detector on the public ship dataset HRSC2016 [59] was validated. The performance comparison results between RAIH-Det (W) and generic models (R${}^2$PN [38], RRD [18], ROI Transformer [19], BBAVectors [20], and R${}^3$Det [60]) on the HRSC2016 dataset are illustrated in

Table 7. Detection results obtained using different general models on the ship dataset HRSC2016.

Model	Backbone	Image Size (Pixels)	AP (%)
RRD [18]	VGG16	$384\times 384$	84.3
R${}^2$PN [38]	VGG16	$500\times 500$	79.6
ROI Transformer [19]	ResNet101	$512\times 512$	86.2
BBAVectors [20]	ResNet101	$600\times 600$	88.2
R${}^3$Det [60]	ResNet50	$600\times 600$	89.26
RAIH-Det (W)	ConvNeXt-T	$600\times 600$	90.64

. The experimental results demonstrate that the proposed method is useful for the detection of arbitrarily oriented ships. We give some detection results of RAIH-Det without head keypoint prediction (“RAIH-Det (W)”) on the HRSC2016 dataset in Figure 11.

6. Conclusions

In this paper, we present an end-to-end rotated aircraft and aircraft head detector (RAIH-Det) that consists of a U-shaped network with ConvNeXt-T for multiscale feature representation and a detection module with extended BBAVectors for mutual information sharing (e.g., head keypoint mutual information). ConvNext-T can facilitate incorporating the relationships among various AIH spatial locations into multiscale feature maps. Our BBAVectors can share more head keypoint mutual information, as it consists of six vectors instead of the original four vectors. This added information can assist in the detection of the head keypoints by exploiting the relationship among the head keypoints, the center point, and the OBB. Additionally, we apply variant CFL, which can focus on more reliable samples, to regress the heatmap locations of the aircraft head keypoints. To facilitate the detection of aircraft and aircraft head, we introduce a DOTA-Plane dataset as a modification of the DOTA-v1.5 dataset. Each aircraft in the new dataset is annotated with the OBB and aircraft head keypoints. Finally, we provide an evaluation and ablation study of the proposed RAIH-Det. The experiments show that the proposed RAIH-Det obtained excellent detection performance improvements on the DOTA-Plane dataset. In the future, the proposed RAIH-Det and the change detection method could be jointed to detect and describe the aircraft changes occurring at an airport using multi-temporal optical remote sensing images. This study can be effectively applied to the real-time monitoring of civilian and military airports. Furthermore, the concept can be extended to multiple tasks. For example, the detection of ships and ship heads in optical remote sensing images is becoming significant in port management, maritime rescue, and maritime and inland freight transport systems.