M³C: Multimodel-and-Multicue-Based Tracking by Detection of Surrounding Vessels in Maritime Environment for USV

Abstract

It is crucial for unmanned surface vessels (USVs) to detect and track surrounding vessels in real time to avoid collisions at sea. However, the harsh maritime environment poses great challenges to multitarget tracking (MTT). In this paper, a novel tracking by detection framework that integrates the multimodel and multicue (M³C) pipeline is proposed, which aims at improving the detection and tracking performance. Regarding the multimodel, we predicted the maneuver probability of a target vessel via the gated recurrent unit (GRU) model with an attention mechanism, and fused their respective outputs as the output of a kinematic filter. We developed a hybrid affinity model based on multi cues, such as the motion, appearance, and attitude of the ego vessel in the data association stage. By using the proposed ship re-identification approach, the tracker had the capability of appearance matching via metric learning. Experimental evaluation of two public maritime datasets showed that our method achieved state-of-the-art performance, not only in identity switches (IDS) but also in frame rates.

1. Introduction

As a sea surface agent, an unmanned surface vessel (USV) needs to independently navigate and perform specific tasks in a maritime environment. Robust and accurate awareness and understanding of the surrounding environment are required to achieve this desired outcome, especially the ability to detect and track surrounding vessels in real time. On a crewed vessel, such critical data can mainly be obtained from navigation radar, sonar, and millimeter wave radar, but these are far from sufficient for a USV, because radar and sonar have minimum detection ranges, and shipborne automatic identification system (AIS) stations are only required for the vessels that weigh over 300 tons, according to the regulations of the International Maritime Organization (IMO) [1,2]. In other words, most of the common sensors have a blind zone at a certain distance, which is precisely what the USV should be paying attention to. Using a vision sensor (visible light or infrared cameras) as auxiliary equipment can compensate for this deficiency.

Approximately 80% of ship and bridge collision accidents are related to human factors, according to the statistical evidence [3]. As deep learning and computer vision have attracted increasing attention, researchers are dedicated to using them to more intuitively performing maritime vessel detection, classification, and tracking tasks. Some maneuvering behavior may occur during collision avoidance, such as acceleration or turning, which must take into account the capricious marine environment, the swing, and yaw imposed by the ego vessel, the change of light, and possible occlusion [4]. All of these pose great challenges to vessel detection and tracking research in the maritime environment. Tracking by detection is one of the state-of-the-art frameworks for multi target tracking (MTT). It is a two-stage strategy, which includes two independent steps: detection and tracking. Detection constructs a solid base for the subsequent tracking. We focus on the latter, on the basis of a state-of-the-art detector.

The significant contributions of this paper can be summarized, as follows. Firstly, we propose a multimodel (MM) filter approach that is based on a gated recurrent unit with attention (GRU-attention) mechanism. The maneuverability of the surrounding vessel is predicted by using the GRU recurrent neural network, being driven by the time series of the historical state of the target vessels. Secondly, a multicue (MC) data association method that considers the long-term and short-term cues is presented. Thirdly, treating the tracking by detection of maritime vessels as a typical problem of monitoring a moving object from video images captured by a moving camera [5], we propose a ship re-identification (Ship-ReID) method, which uses metric learning to determine whether the same ship is in the incoming video frame. We are the first to introduce GRU-attention for surrounding vessel maneuver prediction, to the best of our knowledge. This paper is also the first to introduce Ship-ReID to solve the identity switches (IDS) problem that is caused by camera motion, appearance variation, occlusion, and even blur in the process of maritime vessel detection and tracking.

The structure of this paper is organized, as follows. The related works are introduced in Section 2. Section 3 presents the proposed MM and MC (M³C) tracking by detection pipeline. We evaluate the performance of M³C on the Singapore Marine Dataset (SMD) [1] and PETS 2016 maritime dataset [6] in Section 4. Finally, Section 5 discusses the main conclusions and future work.

2. Related Work

2.1. Deep Learning for Generic Target Detection

The convolutional neural network (CNN) has played an increasingly important role in artificial intelligence, especially in the field of computer vision, as a new structure of neural network. The CNN-based detectors are roughly classified into two categories: two stages and single stage. For the two-stage detection method, the features are first extracted from the image, then proposal regions are generated, followed by classification task performance, and finally, the positioning regression task is completed. The typical algorithms include regions with CNN features (R-CNN), fast/faster R-CNN, R-FCN [7,8,9], and so forth. The single-stage detector omits the task of generating proposal regions in the two-stage method, and such as single shot multibox detector (SSD) [10], YOLOv1/v2/9000/v3 [11] are the typical frameworks. In recent years, anchor-free methods have become very popular, which use the corner or area as the anchor instead of the bounding box directly [12,13].

2.2. Deep Learning for Generic Target Tracking

Traditional target tracking algorithms are divided into tracking by detection and tracking before detection, according to the interaction between the detector and tracker in the tracking process [14]. Tracking before detection can effectively detect dim and small targets. It has been extensively used in radar and infrared image processing. Based on machine vision, especially in visible vision processing, more attention has been paid to joint detection and tracking [15], rather than tracking before detection. Gordon et al. proposed a real-time generic tracker by incorporating temporal information that is founded on a recurrent neural network [16]. Bae et al. proposed an online tracking framework by combining tracklet confidence with deep appearance learning [17]. Arandjelović applied a multiview appearance to track and recognize vehicles in the aerial sequential images [18]. Re-identification is a subissue of image retrieval, which aims to identify the same target from multiple cross cameras or the same target at different moments from a single camera. In the field of using target re-identification to improve the tracking performance, most of the current achievements have focused on pedestrians [19,20] or vehicles [21,22], and there have been no reports that are related to achievements in the detection and tracking of maritime vessels.

2.3. Visible Vessel Detection and Tracking

Despite state-of-the-art object detection and tracking for generic objects having recently demonstrated impressive performances, these have not been adequate for the complicated maritime environment. It is common knowledge that, as surrounding vessels traveling from far to near enter the ego vessel’s surveillance area, they always first appear near the sea antenna (horizon line). With regard to moving shipborne or even buoy-mounted cameras, ship detection and tracking while using the guidance of the sea-sky line has also been proposed in recent years [23]. Jeong et al. proposed a method to estimate the horizon line while using the region of interest [24]. Horizon line detection, dynamic background, and foreground segmentation was performed by means of discrete cosine transform in [25]. Sun et al. also proposed a robustly coarse-fine-stitched strategy to detect the horizon line for USV [26].

Bovcon et al. explored semantics segmentation assisted by an inertial measurement unit (IMU) for stereo obstacle detection [27]. Cane et al. evaluated the semantic segmentation networks on several public maritime datasets and compared their performances [28]. Kim et al. proposed a probabilistic method to detect and classify ships based on a faster R-CNN detector and improved the probability of ship detection by intersection over union (IOU) tracking [29]. Marié et al. proposed a key point tracking method to generate high-quality region proposals and then fed them into a fast R-CNN for further processing [30]. Cao et al. made use of a CNN to extract the features from the vessel image and to eventually identify the ship in a frame of a video sequence [31]. Leclerc et al. took full advantage of ship classification that is based on deep transfer learning to enhance the estimation capability of the trackers [32].

3. The Proposed Tracking by Detection Approach

In this section, we present a novel tracking by detection framework combining MM and MC (M³C), which has the ability to re-identification the vessels that have lost their track. The overall architecture of M³C is detailed in Figure 1.

3.1. The Detector Based on YOLOv3

We devised a maritime vessel detector based on YOLOv3. YOLOv3 is one of the state-of-the-art CNN-based generic object detectors, and it treats the detection and classification of targets as a regression problem [11]. By regressing the location of the detected objects and performing classification, the bounding box and class of the objects are obtained by looking at each video frame only once. It can conduct three different scale predictions. When feeding into a 608 × 608 pixel image, the detection is performed using the scales of 19 × 19, 38 × 38, and 76 × 76, respectively. The structure of the detector is shown in the Figure 2.

3.2. Multimodel for Tracking by Detection.

We formulated the maneuvering prediction of a surrounding vessel as the estimation of the matching probability of each candidate model at the next moment based on the vessel’s historical trajectory and while considering the ego vessel’s posture via GRU recurrent neural networks.

3.2.1. Maneuvering Model of Vessels

According to the kinematic behavior [34,35,36] of maritime vessels, we predefined the three most common kinds of motion maneuvers in advance as candidate models, named constant velocity ($\mathrm{CV}$), constant acceleration ($\mathrm{CA}$), and curvilinear motion ($\mathrm{CM}$), respectively.

The $\mathrm{CV}$ model was used to describe the target performing straight line motions at a constant velocity in the two-dimensional (2D) plane. The state vector describing the dynamic characteristics of the vessel consisted of two elements: pixel position of the target $x,y$ and velocity $\dot{x},\dot{y}$ (i.e., the state vector at time t can be denoted by $X_{CV}(t)={\left[\begin{array}{cccc}{x}_t& {\dot{x}}_t& y_t& {\dot{y}}_t\end{array}\right]}^T$, satisfying the conditional that $\ddot{x}=0,\ddot{y}=0$). The target state equation of $\mathrm{CV}$ at the current time was modeled, as follows:

$$X_{CV}(t+\tau )=F_{CV}(t)X_{CV}(t)+G_{CV}(t)v(t)$$

(1)

where $\tau$ is an indication of the time interval between the current and previous measurement and $v(t)={[v_x,v_y]}^T$ is the process noise vector. The transition matrix $F_{CV}(t)$ and the process noise distribution matrix $G_{CV}(t)$ are stated, as follows, respectively:

$$F_{CV}(t)=\left[\begin{array}{cccc}1& 0& \tau & 0\\ 0& 1& 0& \tau \\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right],G_{CV}(t)=\left[\begin{array}{cc}0.5{\tau }^2& 0\\ 0& 0.5{\tau }^2\\ \tau & 0\\ 0& \tau \end{array}\right]$$

(2)

A vessel modeled as $\mathrm{CA}$ was usually considered to move at constant acceleration. The state vector of a vessel consisted of three components: position $x,y$, velocity $\dot{x},\dot{y}$, and acceleration $\ddot{x},\ddot{y}$ and satisfied the conditional that $\stackrel{⃛}{x}=0$. The target state equation was the same as Equation (1), but the state vector was $X_{CA}(t)={\left[\begin{array}{cccccc}{x}_t& {\dot{x}}_t& {\ddot{x}}_t& y_t& {\dot{y}}_t& {\ddot{y}}_t\end{array}\right]}^T$. The transition matrix $F_{CA}(t)$ and the process noise distribution matrix $G_{CA}(t)$ are stated, as follows, respectively:

$$F_{CA}(t)=\left[\begin{array}{cccccc}1& \tau & 0.5{\tau }^2& 0& 0& 0\\ 0& 1& \tau & 0& 0& 0\\ 0& 0& 1& 0& 0& 0\\ 0& 0& 0& 1& \tau & 0.5{\tau }^2\\ 0& 0& 0& 0& 1& \tau \\ 0& 0& 0& 0& 0& 1\end{array}\right],G_{CA}(t)=\left[\begin{array}{cc}0.5{\tau }^2& 0\\ \Delta t& 0\\ 1& 0\\ 0& 0.5{\tau }^2\\ 0& \tau \\ 0& 1\end{array}\right]$$

(3)

The $\mathrm{CM}$ model was based on $\mathrm{CV}$ and $\mathrm{CA}$, and it depicted the parabolic trajectory of a ship in a turning mode. For this, we needed to estimate the turning rate $\omega$ in real time, and the state vector was extended to $X_{CM}(t)={\left[\begin{array}{cccc}{x}_t& {\dot{x}}_t& y_t& {\dot{y}}_t\omega \end{array}\right]}^T$. The transition matrix $F_{CM}(t)$ and the process noise distribution matrix $G_{CM}(t)$ are stated, as follows, respectively:

$$F_{CM}(t)=\left[\begin{array}{ccccc}1& {\omega }^{-1}\sin \omega \tau & 0& {\omega }^{-1}(\cos \omega \tau -1)& 0\\ 0& \cos \omega \tau & 0& -\sin \omega \tau & 0\\ 0& {\omega }^{-1}(1-\cos \omega \tau )& 1& {\omega }^{-1}\sin \omega \tau & 0\\ 0& \sin \omega \tau & 0& \cos \omega \tau & 0\\ 0& 0& 0& 0& 1\end{array}\right],G_{CM}(t)=\left[\begin{array}{ccc}0.5{\tau }^2& 0& 0\\ \tau & 0& 0\\ 0& 0.5{\tau }^2& 0\\ 0& 0& 0\\ 0& 0& 1\end{array}\right]$$

(4)

3.2.2. Encoder-Decoder Model of GRU-Attention

We utilized the multimodel weighting method to adapt the change of the target stateI in our proposed M³C pipeline, during the kinematic filter procedure. The primary task was to estimate the match score of each predefined candidate model and the current state of the vessels. For this, the GRU-attention model was deployed to improve the prediction performance. Long short-term memory (LSTM) networks were proposed to solve the long-term dependencies problem of traditional recurrent neural networks (RNNs) with increasing time intervals [37]. Through proper training, it can remember the critical data and forget the less-important data. GRU [38] improves LSTM by reducing the number of gates and removing all of the memory units, so that only reset gates and update gates remain.

As shown in Figure 3, the formulas for forward propagation of GRU are as follows:

$$\begin{array}{l}{z}_t=\sigma (w_r\cdot [h_{t-1},x_t]+b_z)\\ r_t=\sigma (w_z\cdot [h_{t-1},x_t]+b_r)\end{array}$$

(5)

where $x_t$ is the input vector at time t, $\sigma$ represents the activation function, $w_r,w_z$ are the weight matrix, $h_{t-1}$ represents the hidden activation value at the previous moment, and $b_z,b_r$ is the deviation vector. The activation value $h_t$ and the candidate activation value ${\tilde{h}}_t$ of the hidden node at time t are calculated, as follows, where $\otimes$ indicates the element-wise multiplication:

$$\begin{array}{l}{\tilde{h}}_t=tanh(W\cdot [r_t\otimes h_{t-1},x_t]+b_h)\\ h_t=(1-z_t)\otimes h_{t-1}+z_t{\tilde{h}}_t\end{array}$$

(6)

As stated above, the curvilinear motion model of vessels can be divided into three subpatterns, according to tangential or normal acceleration: $\mathrm{CV},\mathrm{CA}$, and $\mathrm{CM}$, respectively. Their coexistence and the uncertainty of switching time leads to an alignment problem. It is very difficult for a traditional LSTM/GRU-based encoder–decoder approach to solve the alignment problem. We introduced an attention model to encode each subsequence into a context vector, instead of encoding the entire sequence into an integral vector. Through training, the attention model was encouraged to selectively focus on the important part of all information while also ignoring other secondary information, so as to generate more accurate prediction results by making full use of the subpatterns.

If we were to estimate the maneuvering model of the surrounding vessel at time t, the input sequence would be $z_{t-N\tau },\dots ,z_{t-\tau },z_t,z_{t+\tau }$, as can be seen from Figure 4. The context vector was the weighted sum of output vector $h_{t-N\tau },\dots ,h_{t-\tau },h_t,h_{t+\tau }$ from encoders, and weights $c_{t-N\tau },\dots ,c_{t-\tau },c_t,c_{t+\tau }$ could be calculated by

$$c_i={\displaystyle \sum _{j=t-N\tau }^{t+\tau }{h}_j{\beta }_{ij}}$$

(7)

where ${\beta }_{ij}$ is the attention weights to be learned, as obtained by the following formula:

$${\beta }_{ij}=\frac{Exp(e_{ij})}{{\displaystyle \sum _{k=t-N\tau }^{t+\tau }Exp(e_{ik})}},s.t.{\displaystyle \sum _j{\beta }_{ij}}=1$$

(8)

$$e_{ij}=FC(s_{i-1},h_j)$$

(9)

The GRU-attention model was fed with a time-series vector, which described the short-term historical motion measurements of the surrounding vessels. It could output the probability that each candidate model matched the current motion state of the target vessels.

3.2.3. Multimodel Filter

As mentioned before, we defined a model set that meets the needs of tracking moving vessels on the sea: $ℳ={\left\{m_i\right\}}_{i=1,2,\cdot \cdot \cdot ,N}$, consisting of a total of N candidate models. If at time $t$ we have already obtained the historical cumulative observation vectors, the state estimation ${\mathcal{Y}}_i^{(t+\tau )}$ of the i-th model $m_i$ can be modeled as the following formula [39], which rests on the conditional probability theory:

$$\begin{array}{cc}\hfill {\widehat{\mathcal{Y}}}_i^{(t+\tau )}& =\mathbb{E}({\mathcal{Y}}^{(t+\tau )}|m_i^{(t+\tau )},Z)\hfill \\ & \triangleq {\mathcal{Y}}^{(t)}+K^{(t)}[X(t+\tau )-F(t+\tau ){\mathcal{Y}}_i^{(t)}]\hfill \end{array}$$

(10)

where $K^{(t)}$ represents the gainer of the Kalman filter, and $m^{(t)}\triangleq P(m^{\left(t\right)}|Z^{(0)})={\psi }^{(t)}(0)$ is the prior probability of the initial time that satisfies the sum-to-one condition of ${\displaystyle \sum _{t=1}^N{\psi }^{(t)}(0)}=1$.

As shown in Figure 4, $Z=\left\{z_{t-N\tau }\dots z_{t-\tau }{z}_t{z}_{t+\tau }\right\}$ as the input sequence, which indicates the cumulative set of measurements until time t. Among $z_j={[u_j,v_j,h_j,{\gamma }_j,{\dot{h}}_j,{\dot{\gamma }}_j,{\dot{x}}_j,{\dot{y}}_j,{\rho }_j,\vartheta ]}^T$, the components of $z_j$ are the state’s parameter of j time, which we separately explain below: $(u,v)$ is the bounding box center coordinate, $h$ is the bounding box height, $\gamma$ represents the bounding box ratio, ${\dot{h}}_j,{\dot{\gamma }}_j$ are the velocities, $({\dot{x}}_j,{\dot{y}}_j)$ are the velocities along the x-axis and y-axis on the 2D plane, $\rho$ represents the root-mean-square (RMS) values of the ego vessel’s pitch and roll, and $\vartheta$ denotes the type of target vessel.

The sums of the multimodel were weighted according to their matched likelihood probability. The motion state of the target vessel was calculated by

$$\begin{array}{cc}\hfill {\widehat{\mathcal{Y}}}^{(t+\tau )}& =\mathbb{E}[{\mathcal{Y}}^{(t+\tau )}|Z]\hfill \\ & ={\displaystyle {\sum }_{i=1}^N{\widehat{\mathcal{Y}}}_i^{(t+\tau )}P(m_{{}^i}|Z)}\hfill \end{array}$$

(11)

and the estimation error of covariance matrix was:

$$\mathcal{P}(t+\tau )={\displaystyle {\sum }_{i=1}^N\mathbb{P}(m_{{}^i}|Z)\{{\mathcal{P}}_i(t+\tau )+[{\widehat{\mathcal{Y}}}_i^{(t+\tau )}-{\widehat{\mathcal{Y}}}^{(t+\tau )}]{[{\widehat{\mathcal{Y}}}_i^{(t+\tau )}-{\widehat{\mathcal{Y}}}^{(t+\tau )}]}^T\}}$$

(12)

where $\mathbb{P}(m_{{}^i}|Z)$ is the posterior probability that can be obtained from the softmax layer of the GRU-attention model with loss and as the individual model’s output:

$$\mathbb{P}(m_i|Z)=\begin{array}{c}\frac{Exp(m_{{}^i})}{{\displaystyle {\sum }_{k=1}^{N}Exp(m_{{}^k})}}\end{array}$$

(13)

The adaptive moment estimation (Adam) was used as the optimizer and the cross entropy for the loss function. We assumed that $\Theta$ represented the parameter set of GRU to be trained, and the loss function was written, as follows:

$$\mathcal{J}(\Theta )=-\frac{1}{N}{\displaystyle \sum _{i=1}^N{m}_{{}^i}Log(h_{\Theta }(m_{{}^i})+(1-m_{{}^i})Log(1-h_{\Theta }(m_{{}^i}))}$$

(14)

3.3. Multicue for Data Association

During the data association procedure, we propose a hybrid affinity model that is based on multi cues to evaluate the similarity between the detected vessels and existing tracklets, which contains both long-term cues and short-term cues. We regard appearance as a long-term cue and, meanwhile, the short-term cues consist of surrounding vessels motion measurements and the dynamic attitude of ego vessel, such as pitch and roll. We use adaptive association gate of appearance to confirm the validation of measurements before the data association algorithm is carried out.

3.3.1. Adaptive Association Gate of Appearance

The simple camera model [40] has been applied to put targets into a three-dimensional (3D) perspective considering the height of the target in the image, and it assumes that all the objects of interest rest on the ground plane. For a vessel sailing on the seaplane, which perfectly coincided with the assumptions, we additionally assumed that the ship’s heading angle was consistent with its course.

Assume that camera parameters $\Theta$ consist of the following elements: focus length $f_{\Theta }$, height $h_{\Theta }$, camera tilt angle ${\psi }_{\Theta }$, absolute velocity ${\upsilon }_{\Theta }$, image center ${\mu }_c$, horizon position $v_c$, and 3D location $(x_{\Theta },z_{\Theta })$. Thus, the projection $f$ can be defined as

$$\tilde{Z}=\left[\begin{array}{cc}R({\psi }_{\Theta })& 0\\ 0& 1\end{array}\right]Z+\left[\begin{array}{c}{x}_{\Theta }\\ z_{\Theta }\\ 0\end{array}\right],X=f(\tilde{Z})={\left[\frac{f_{\Theta }{x}_Z}{z_Z}+u_c\frac{f_{\Theta }{h}_{\Theta }}{z_Z}+v_c\frac{f_{\Theta }{h}_Z}{z_Z}\right]}^T$$

(15)

where $X=[\mu ,v,h]$ represents the central coordination and the height of the vessel in the image plane, while $\tilde{Z}$ denotes the location in the current camera coordinates. We could easily estimate the scale and location variances of the vessel in the image based on the simple linear relationship between the vessel’s image size and the y-axis image location [41].

When the target vessel moves closer to the camera, the bounding box of imaging area gradually becomes larger and vice versa. Suppose that the bounding box center of the current target vessel is at (x, y) coordinate of the image, x and y divide the whole image into four quadrants, and each quadrant is a candidate region. We present four possible situations in Figure 5, while considering the pitching and rolling attitude of the ego vessel at the present moment.

3.3.2. Long-Term Cues

It is well known that the condition of a vessel on the sea is much more complex than that of vehicles and pedestrians on the ground due to the influence of environmental factors, such as waves, wind, and ocean currents. The camera also moves with the movement of the ego vessel, such as swing (surge, sway, and heave) and rotating (roll, pitch, and yaw), which cause the target to be temporarily lost from the camera scene. These dramatic changes on sea surface require the re-identification of the same vessel from adjacent frames or even across some frames according to their similarity measurements as the long-term cues.

The MDNet structure proposed in [22] was employed here, and a dataset with more than 500 vessels and nearly 150,000 images, named Ship-ReID, was constructed. We found the optimal mapping from image space to feature space after training on it. Suppose that inputs are triplet units $\left\{<{ℐ}_a,{ℐ}_p,{ℐ}_n>\right\}$, indicating the anchor image, positive image, and negative image, respectively. $<{ℐ}_a,{ℐ}_p>$ is a positive pair belonging to the same vessel, while $<{ℐ}_a,{ℐ}_n>$ is the negative pair from different vessels. We used the normalized cosine similarity between the feature vector to measure the distance, such as ${\mathcal{D}}_{a,p}$ and ${\mathcal{D}}_{a,n}$, and guided the network to train in the direction of pushing positive and negative samples away. We ensured that the distance between positive samples was very close by means of the improved triplet loss, as follows:

$$ℒ({ℐ}_a,{ℐ}_p,{ℐ}_n)={\displaystyle \sum ^N\left[{\mathcal{D}}_{a,p}+max({\mathcal{D}}_{a,p}-{\mathcal{D}}_{a,n}+\lambda ,0)\right]}$$

(16)

where $\lambda$ is the pre-set threshold parameter.

3.3.3. Data Association and Fusion Method

As introduced in [42], we adopted heuristic simulated annealing to solve the assignment problem during the data association stage [43]. Assuming that we have $D$ detectors and $T$ tracklets waiting to match at time $k$, and $\mathcal{D}=\{d_1,d_2,\cdot \cdot \cdot ,d_D\},\mathcal{T}=\left\{t_1,t_2,\cdot \cdot \cdot ,t_T\right\}$, ${\mathcal{G}}^{(k)}$ is the adaptive gate association set, and ${ℱ}^{(k)}$ is the tracking gate set consisting of the rectangular region generated by the coordination predicted by MM filter trackers. Subsequently, the association cost matrix between the $i$-th tractlet’s predictions ${\widehat{t}}_i^{(k)}$ and the $j$-th detectors $d_j^{(k)}$ could be defined as

$$C^{(k)}(i,j)=\{\begin{array}{c}\begin{array}{c}{c}^{(k)}({\widehat{t}}_i,d_j),if d_j^{(k)}{\displaystyle \cap {\mathcal{G}}_i^{(k)}\ne \varnothing or d_j^{(k)}{\displaystyle \cap {\mathcal{G}}_i^{(k)}\ne \varnothing }}\\ \infty , \mathrm{otherwise}\end{array}\end{array}$$

(17)

where $c({\widehat{t}}_i,d_j)$ is the hybrid affinity metric of association at time $k$ when the $j$-th detector falls into the correlation gate corresponding to the prediction coordinate of the $i$-th tracklet, which can be calculated as

$$c^{(k)}({\widehat{t}}_i,d_j)=(1-\mathcal{A})\cdot motio{n}^{(k)}({\widehat{t}}_i^{(k)},d_j^{(k)})+\mathcal{A}\cdot appea{r}^{(k)}({\widehat{t}}_i^{(k)},d_j^{(k)})$$

(18)

where the weight coefficient is denoted as $\mathcal{A}$, which is depicted by the attitude of the ego vessel. Equation (18) demonstrates that, as the pitch and roll of the hull increase, the weight of the appearance is enhanced, and vice versa.

$$\mathcal{A}=\frac{{\left[{(p^{(k)})}^2+{(r^{(k)})}^2\right]}^{1/2}}{\lambda {\left[{(p_{max})}^2+{(r_{max})}^2\right]}^{1/2}}$$

(19)

where $p,r$ is the dynamic pitch and roll at time k, which can be obtained from real-time measurement of the ego vessel’s electric compass or IMU; $p_{max},r_{max}$ represent the maximum range of pitch and roll, which are determined by the intrinsic characteristics of the hull, which can be obtained from prior measurement; and, $\lambda$ is a hyper parameter.

$appear({\widehat{t}}_i,d_j)$ is the measurement of appearance similarity between ${\widehat{t}}_i$ and $d_j$, and $motion({\widehat{t}}_i,d_j)$ represents their measurements of motion similarity that were obtained by calculating the negative logarithm of the intersection-of-union ratio:

$$motion({\widehat{t}}_i^{(k)},d_j^{(k)})=-log\left(\frac{Intersection({\widehat{t}}_i^{(k)},d_j^{(k)})}{Union({\widehat{t}}_i^{(k)},d_j^{(k)})}\right)$$

(20)

3.4. The Proposed M³C Tracking by Detection

In this section, an end-to-end and real-time tracking by detection pipeline is presented, for which the tracklet initiation and termination conditions were both determined by a continuous three-frame method, as described in Algorithm 1:

Algorithm 1: The proposed M3C tracking by detection

Inputs: The sequential frames of surrounding vessels $\mathcal{S}$, pre-trained GRU model ${\mathcal{M}}_{gru}$, Darknet53 model ${\mathcal{M}}_{d53}$, Ship-Reid model ${\mathcal{M}}_{reid}$, and periodic attitude data $\mathcal{A}$ of the ego vessel

Initialization: $\mathcal{T}\leftarrow \varnothing$, ${\mathcal{A}}_{max}$, ${(v_k)}_{count}=0$, ${(t_i)}_{count}=0$

Outputs: Continuous tracklets $\mathcal{T}={\left\{t_i\right\}}_{i=1}^M$ of surrounding vessels

Procedure:

foreach frame at current time $t$

detect all potential vessels $\mathcal{V}={\left\{v_k\right\}}_{k=1}^N$ from the ${ℱ}_t$ frame using ${\mathcal{M}}_{d53}$

foreach tracklet of vessel  // Association gate generated from prior time

Predict the kinematic track gate from MM filter from time $t-1$

& adaptive association gate of appearance

endfor

MC data association using simulated annealing algorithm

foreach pair of matched detection and tracklet

tracklet update by MM filter

endfor

foreach detection vessel $v_k$ not associated with any tracklets in $T$

${(v_k)}_{count}++$

if ${(v_k)}_{count}>=3$  //Three-frame tracklet initiation

initialize a new tracklet $t$

$\mathcal{T}\cup =\left\{t\right\}$

end

endfor

foreach vessel tracklet $t_i$ not associated with any detection in $V$

${(t_i)}_{count}++$

tracklet extrapolation

if ${(v_k)}_{count}>={\lambda }_{max}$  // Extrapolation times exceeding the pre-set value ${\lambda }_{max}$

terminate the tracklet $t_i$

$\mathcal{T}=\mathcal{T}-\left\{t_i\right\}$

end

endfor

endfor  //reach the end of the video sequence

4. Experiment and Results Analysis

4.1. Experimental Setup

4.1.1. Datasets

We performed extensive experiments on SMD [1] and the PETS 2016 maritime dataset [6] to evaluate performance of the proposed M³C. The detector was pretrained on a Marvel vessel image dataset by the offline method [44].

SMD contains 51 annotated video fragments, 40 onshore videos, and 11 onboard videos. We estimated the approximate range of roll and pitch of the ego vessel from horizon ground truths (GTs). The PETS 2016 maritime dataset includes 20 RGB video fragments that were captured by four digital pan-tilt-zoom (PTZ) cameras, three of which were at the side and one at the stern. Two skiff boats and two fish boats as the supplementary target vessels performed behaviors, such as speeding up, loitering, moving around the ego vessel, and so on. The collected the Marvel vessel image dataset that consisted of more than 140,000 vessel images. It was divided into 26 superclasses, drawing from the www.shipspotting.com website of online vessel photos and trackers. We resized the image to 608 × 608 pixels and manually labeled all of the selected pictures as belonging to five superclasses, which were tug ship, container, fish boat, skiff boat, and passenger ship.

4.1.2. Implementation Details

We used an i7-8700K CPU with 32 GB of memory and dual NVIDIA Titan RTX GPU with a Pytorch deep learning framework to implement our algorithm. We devised the detector based on the state-of-the-art YOLOv3, as mentioned in the previous section. The size of prior anchors is list in

Table 1. The feature map and prior anchors of Singapore Marine Dataset (SMD) and PETS 2016 maritime dataset. A total of nine anchors were used to generate bounding boxes, and the size of prior anchors was calculated by the k-means algorithm.

Feature Map	19 × 19			38 × 38			76 × 76
Receptive Field	Big			Medium			Small
Prior Anchors	110 × 84	176 × 93	267 × 166	35 × 65	62 × 57	107 × 39	43 × 19	38 × 39	68 × 29

.

We used GRU with 128 units for the vessel’s maneuverability discrimination and trained the model while using Adam with a dropout of 0.2 for regularization and with a learning rate of 0.001. Due to the lack of adequate vessel tracklets or video galleries for training, it was pretrained by the public AIS dataset [45], and was then fine-tuned on the training set partitioned from SMD and PETS 2016.

4.2. Performance Evaluation

4.2.1. Evaluation Metric

The evaluation was carried out according to the standard root mean square error (RMSE)/mean absolute deviation (MAE) and CLEAR MOT metrics that were proposed in [46], and a brief description of each metric is listed below:

MOTA (↑): multi-object tracking accuracy, as calculated by the following formula:

$$MOTA=1-\frac{{\displaystyle {\sum }_t(N_{FN}^{(t)}+N_{FP}^{(t)}}+N_{IDS}^{(t)})}{{\displaystyle {\sum }_t{N}_{GT}^{(t)}}}$$

(21)

where $N_{FN}^{(t)}$, $N_{FP}^{(t)}$, and $N_{IDS}^{(t)}$ are the number of false negatives, false positives, and IDS in the t frame index; and, $N_{GT}^{(t)}$ represents the number of ground truth targets.

MOTP (↑): multi-object tracking precision, as calculated by the following formula:

$$MOTP=\frac{{\displaystyle {\sum }_{i,t}{d}_i^{(t)}}}{{\displaystyle {\sum }_t(N_{TP}^{(t)}+N_{IDS}^{(t)})}}$$

(22)

where $N_{TP}^{(t)}$ is the number of true positives in the t frame index and $d_i^{(t)}$ represents the bounding box overlap of the i-th target.

RMSE (↓): root-mean-square error; MAE (↓): mean absolute deviation; MT (↑): mostly tracked targets; ML (↓): mostly lost targets; FN (↓): total number of false negatives; FP (↓): total number of false positives; IDS (↓): identity switches; FPS (↑): frames per second; and, mAP (↑): mean average precision. The colored arrow after each metric index indicates whether the value of increasing (↑) or decreasing (↓) is beneficial. MOTP and MOTA lie in the range of [0,100%], with the best value being 100%.

4.2.2. Qualitative Results

For the sake of fairness, performance comparison was separately carried out according to onshore or onboard datasets, because the results of some trackers can be biased due to camera motion. We discuss the comparison with state-of-the-art tracking methods, such as the framework of Markov decision process (MDP) [47], the combination of Kalman filter and Hungarian assignment algorithm (SORT) [48], the kernel correlation filter method (KCF) [49], the combination of Kalman filter and Kuhn-Munkres (POI) [50], SORT with deep association metric (DeepSORT) [19], and candidate selection combined with re-identification (MOTDT) [51].

The M³C pipeline obtained a 2.5% increase and a 33.5% decrease in MOTA and IDS as compared with the second best onshore dataset, as shown in

Table 2. Quantitative evaluation of different trackers on the two maritime datasets. We treated PETS 2016 as the onshore dataset considering a stationary large ship.

Dataset	Tracker	MOTA↑	MOTP↑	MT↑	ML↓	IDS↓	FPS↑
SMD (onshore) + PETS 2016	MDP	30.3%	71.3%	13.2%	38.4%	426	1
	SORT	59.8%	79.6%	25.4%	22.7%	631	56
	KCF	70.3%	80.2%	37.8%	22.3%	382	25
	POI	66.1%	79.5%	34.6%	20.8%	453	10
	M3C (Ours)	72.8%	80.4%	37.4%	21.2%	254	20
SMD (onboard)	DeepSORT	60.4%	79.1%	32.8%	18.2%	56	36
	MOTDT	57.6%	70.9%	34.2%	28.7%	49	23
	M3C (Ours)	63.4%	74.6%	26.2	17.9%	34	16

. As far as the onboard dataset, the corresponding figures were 3% and 30.6%, and ML was also the lowest. In addition, on both onshore and onboard datasets, FPS was more than 10 in both scenarios, which indicated that real-time performance was obtained.

4.2.3. Visual Tracking Results

We present the visual tracking results of some videos to illustrate the performance of our proposed M³C framework intuitively.

In Figure 6 and Figure 7, we show the tracking robustness of a reappearing vessel after being lost for a short time Figure 7b, and even complete occlusion Figure 6b, for which our proposed Ship-ReID worked well as the long term cue.

4.3. Ablation Study and Analysis

4.3.1. Effect of Multimodel Fusion Filter

In this scheme, we considered three different motion scenarios of the target vessel, including a straight line with constant velocity motion, acceleration, and turning motion. We conducted an ablation experiment by replacing MM with a single Kalman filter ($\mathrm{CV}$) and interactive multiple models (IMM) [52] while considering the $\mathrm{CA}$, $\mathrm{CV}$, and $\mathrm{CM}$ model by keeping other conditions unchanged. RMSE and MAE were used as the evaluation indicators, respectively.

Table 3. Results of averaged RMSE and mean absolute deviation (MAE). SMD (onshore) and the PETS 2016 dataset were split into three different motion scenarios for the experiment, and a Kalman filter with a single constant velocity ($\mathrm{CV}$) model was used as the benchmark.

Motion Scenarios	Algorithm	Averaged RMSE (L2 Norm)		Averaged MAE (L1 Norm)
		X pos.	Y pos.	X pos.	Y pos.
Straight line, constant velocity	Kalman ($\mathrm{CV}$)	14.71	16.06	11.47	14.53
	IMM ($\mathrm{CV}+\mathrm{CA}+\mathrm{CM}$)	15.10	15.87	12.03	14.28
	MM(Proposed)	9.10	9.58	5.75	6.02
Acceleration	Kalman ($\mathrm{CV}$)	27.71	21.29	25.38	20.54
	IMM ($\mathrm{CV}+\mathrm{CA}+\mathrm{CM}$)	21.17	18.26	18.67	15.88
	MM(Proposed)	13.93	12.21	8.04	7.04
Turning	Kalman ($\mathrm{CV}$)	20.99	15.85	19.54	15.03
	IMM ($\mathrm{CV}+\mathrm{CA}+\mathrm{CM}$)	15.10	13.11	12.93	10.98
	MM (Proposed)	12.44	9.56	8.80	6.67

shows the comparison results.

IMM did not perform better than the single $\mathrm{CV}$ model with a Kalman filter when no maneuver occurred and was even worse than the latter, because of the possible delay of model transfers, as can be seen in Table 3. The proposed MM filter overcame this very well, and RMSE and MAE both increased by 30% in the three common motion scenarios.

4.3.2. Effect of Detector

As illustrated in

Table 4. Detector performance in the ablation study. We replaced the detector with SSD and faster regions with CNN features (R-CNN), respectively, for these ablation experiments.

Tracker	Detector	mAP	FPS	Averaged RMSE (L2 Norm)		Averaged MAE (L1 Norm)
				X pos.	Y pos.	X pos.	Y pos.
M3C	SSD300 (Mobilenet)	41.2	46	20.22	17.59	9.54	16.37
	YOLOv3 (Darknet53)	57.9	20	9.10	9.58	7.75	6.02
	Faster R-CNN (Resnet50)	59.1	4	8.94	6.39	8.44	5.83

, the detector built on YOLOv3 with Darknet53 as the backbone increased mAP by nearly 40% when compared with SSD, and FPS was five times higher than that of faster R-CNN. It was a good compromise between mAP and FPS, and both of them were equally critical for improving tracking performance. These experiments showed that the quality of the detection algorithm has significant influence on multi target tracking performance.

4.3.3. Effect of Multicue Data Association

It can be observed from

Table 5. Results of the ablation study. multimodel (MM), single model (SM), and re-identification (ReID) denote appearance model, multimodel filter, single model filter, Ship-ReID, respectively.

Tracker	Detector	MOTA↑	MOTP↑	MT↑	ML↓	IDS↓	FPS↑
MM	YOLOv3	46.2%	44.5%	12.9%	43.2%	74	28
MM + Attention	YOLOv3	47.1%	43.8%	13.1%	44.7%	73	32
SM+ReID	YOLOv3	59.8%	64.5%	21.4%	23.6%	62	20
MM+ReID (M3C)	YOLOv3	63.4%	74.6%	36.2%	17.9%	24	16

that the M³C pipeline obtained the highest value of MOTA, which is one of the most important indicators of multitarget tracking. These data lead us to the conclusion that combining the motion features with appearance features can effectively improve the tracking accuracy.

By combining deep feature extraction and a traditional kinematic filtering algorithm, the proposed M³C approach strikes a balance between tracking accuracy and speed and improves tracking accuracy while guaranteeing real-time performance, as discussed previously.

5. Conclusions

We proposed a novel tracking by detection approach integrating MM and MC to detect and track surrounding vessels of USVs at sea in this paper. MM was used to solve the problem of unstable tracking of a maneuvering target in the traditional single-model Kalman tracker (such as the $\mathrm{CV}$ model). MC combines the attitude of the ego vessel and the appearance of the target vessels to solve the problem of frequent IDS that is caused by motion blurring and occlusion. Experiments have demonstrated its efficiency and robustness and it achieved real-time performance. In the future, we plan to focus on the following two directions. First, we will construct a more complete Ship-ReID dataset and optimize the network framework and algorithm. Second, we intend to integrate Ship-ReID with a convolutional neural network of the detector and to co-train them in a unified network, so as to further improve the real-time performance.