Cell-Based Target Localization and Tracking with an Active Camera

Abstract

This paper proposes a new method of target localization and tracking. The method consists of four parts. The first part is to divide the scene into multiple cells based on the camera’s parameters and calibrate the position and error of each vertex. The second part mainly uses the bounding box detection algorithm, YOLOv4, based on deep learning to detect and recognize the scene image sequence and obtain the type, length, width, and position of the target to be tracked. The third part is to match each vertex of the cell in the image and the cell in the scene, generate a homography matrix, and then use the PnP model to calculate the precise world coordinates of the target in the image. In this process, a cell-based accuracy positioning method is proposed for the first time. The fourth part uses the proposed PTH model to convert the obtained world coordinates into P, T, and H values for the purpose of actively tracking and observing the target in the scene with a PTZ camera. The proposed method achieved precise target positioning and tracking in a 50 cm ∗ 250 cm horizontal channel and a vertical channel. The experimental results show that the method can accurately identify the target to be tracked in the scene, can actively track the moving target in the observation scene, and can obtain a clear image and accurate trajectory of the target. It is verified that the maximum positioning error of the proposed cell-based positioning method is 2.31 cm, and the average positioning error is 1.245 cm. The maximum error of the proposed tracking method based on the PTZ camera is 1.78 degrees, and the average error is 0.656 degrees.

1. Introduction

With the continuous advancement of science and technology and the continuous improvement of people’s living standards, video surveillance has played an increasingly important role in people’s lives [1,2,3,4]. The traditional monitoring method based on a fixed perspective can no longer meet people’s increasing demand for security protection. Therefore, PTZ cameras with rotation, pan, and zoom functions have gradually become a focus of people’s attention [5,6,7,8]. A camera with a PTZ function tracks and observes the target in the scene by changing its own parameters. However, due to the constantly changing position, speed, size, and background of the moving target in the scene, it is easy to cause a situation where the target cannot be accurately located, or the tracking target is lost. Recently, domestic and foreign researchers have conducted many related studies, but there are still many key issues to be urgently solved [9].

The PTZ camera has the function of real-time perception of the external characteristics of the target in the scene. However, if one wants to further perceive other target characteristics, such as type, location, and speed, a new observation model must be built.

To use the PTZ camera to obtain accurate target information such as type, position (speed), etc., we propose a target localization and tracking method based on cell and active cameras. In this method, we divide the area of target movement into multiple cells. We then use deep learning-based algorithms to identify the vertices and targets of the cells in the scene, and then use the PnP algorithm to calculate the world coordinates of the targets in the image. Finally, based on the target’s world coordinates, the PTH model is used to calculate the P and T values of the PTZ camera, in order to achieve the purpose of tracking and observing the target.

This experiment was conducted with a wireless remote-control Jeep (hereafter referred to as “WRC-Jeep”) as the target and it was compared with other target tracking methods. The main contributions of this article are summarized as follows:

(1) A cell segmentation and recognition method is proposed for the first time. This method divides the experimental scene into multiple cell areas, calibrates each cell’s vertices, and then uses the YOLOv4 detection algorithm to identify the vertices of the cells and the target in the image. Finally, the proposed vertex matching method is used to match the vertex in the image and vertex in the scene.

(2) A method for accurately calculating the target position is proposed first. Firstly, we used the PnP model to calculate the target’s initial position. Secondly, we used the proposed error model to calculate the target error position. Finally, we used the target position to subtract the error position to obtain the precise target position.

(3) A PTH model is constructed. The world coordinates can be transformed into P value, T value and H value based on the PTZ camera.

The rest of this paper is organized as follows. Section 1 surveys works that are closely related to our method. Section 2 presents the proposed method in detail. The experimental results are demonstrated in Section 3. Finally, Section 4 concludes the paper.

2. Related Work

2.1. Image-Level Target Tracking

Before 2010, researchers achieved fruitful results in the field of target tracking. Classic target tracking algorithms included Meanshift [10], Camshift [11], particle filter [12], and the Kalman filter algorithm, as well as the optical flow algorithm, and so on. However, these classic target tracking algorithms also have some problems, including limitations in feature representation. When the target is deformed, partially occluded, and there is similar target interference in the background, it will cause the tracking target to be lost, or cause a large error in the target trajectory.

The Siamese network trackers and correlation filter trackers have better performance, with the appearance of many new target tracking algorithms. Compared with the traditional correlation filtering algorithm, when the SiamFC algorithm [13] is used to track the targets, the targets in the first frame are used as the templates for subsequent frame tracking, and the templates will not change. This also causes the tracking template not to change with the change of the target. The disadvantage is that it cannot perceive the change of the target. When the parameters of the target change, the tracked target is likely to be lost. Classical Siamese network tracking algorithms include SiamFC [13], SiamRPN [14], SiamRPN++ [15], etc.

The essence of the correlation filtering algorithm and its improved algorithm [16,17,18,19,20,21] is to find an area with the greatest correlation with the template in the current frame of the video sequence. The tracking template comes from a filter of online learning. In tracking the target, the correlation filtering algorithm continuously updates the template. The performance of the updated template directly determines the effect of target tracking. The classic correlation filtering algorithms include MOSSE [22], CSK [23], KCF [24], DSST [25], SAMF [26], SRDCF [27], Staple [28], C-COT [29], ECO [30], and ATOM [31].

Many excellent deep learning-based target tracking algorithms have emerged. They are divided into two categories. One is based on region proposals, and the main representatives are R-CNN, SPP-net, Fast R-CNN, Faster R-CNN, R-FCN, etc. The others are based on the end-to-end algorithm, mainly represented by SSD [32] and the YOLO algorithm [33,34,35,36].

2.2. Device-Level (PTZ Camera) Target Tracking

Hu et al.’s [37] method uses two PTZ cameras to observe moving targets in the path, such as alternate channels. The experimental scene is divided into several continuous cells, and then each cell is calibrated. The PTZ camera quickly calibrates online after the operation and then tracks the moving targets in the cell for observation.

In Nacer Farajzadeh et al.’s method [38], the area of interest is first manually selected, and then the three features of the feature points in the area are extracted. The features include the mean, the variance, and the range of intensity values. Then, they calculate the transformation matrix with the information of the feature points and the K-means algorithm for the purpose of tracking the target of interest in the image.

Liu et al. [39] proposed a target tracking model based on a PTZ camera. The model consists of background modeling, observation frame registration, and target tracking. The first step is to build a background model for the observation scene. Then, the feature extraction algorithm is used to register the region of interest in the adjacent frames; finally, the classic target tracking algorithm is used to cooperate with the PTZ camera to achieve the purpose of actively tracking the target in the scene. This method makes full use of the advantages of the PTZ camera and combines the image-level target tracking algorithm to realize the tracking of the scene target.

Kang et al. [40] built a moving target tracking system based on a PTZ camera. The system includes three parts: adaptive background generation, moving region extraction and tracking. Similar to the method proposed by Liu et al., it also needs to build a background model of the scene first, and then extract the moving target from the continuous frame. Finally, we changed the PTZ parameters of the active camera for the tracking target.

Lisanti et al. [41] proposed an online calibration method for a PTZ camera. First, the internal and external parameters of the camera were calibrated offline before the operation. Then, when the PTZ camera started working, the shooting attitude of the current camera was calculated by extracting the captured image features, and then it was compared with the external parameters of the calibrated camera. Following these steps, the camera’s attitude was adjusted to track and observe the target.

Shakeri et al. [42] used a dual-camera system to locate and track small targets in the scene. This method constructs a camera model that extracts the target features to calculate the best posture. We adjusted the parameters of the PTZ camera to track and observe the target.

Baris et al. [43] constructed a dual-camera system for target tracking and observation. The dual-camera system includes an omnidirectional camera and a PTZ camera. First, the omnidirectional camera provides a 360-degree field of view to locate and classify the target. Then, the PTZ camera is used to track the target.

Hu et al. [44] constructed a PTZ camera observation model, namely a four-cell-based image target tracking method. They achieved image target tracking by one PTZ camera. However, the method cannot precisely locate the target and has a significant tracking error.

3. Methodology

The proposed method mainly includes four parts. Firstly, the scene is divided into multiple cells that were calibrated; secondly, YOLOv4 is used to detect the image coordinates of the target and the cells’ vertices. Then, the cells’ vertices in the image are matched by the proposed matching method; thirdly, the precise world coordinates of the target are calculated by the proposed method; finally, the world coordinates are transformed into angles of the PTZ camera, to track and observe the target. The overall flow chart of the system is shown in Figure 1. The specific method is to change the P value and T value of the PTZ camera continuously to ensure that the moving target is always in the center of the FOV (field of view).

3.1. Cell Division and Vertex Calibration in the Experimental Scene

In a wide traffic scene, it is often impossible to observe the dynamics of the entire scene through a fixed-view camera. In contrast, a PTZ camera obtains information about targets in a wider scene by adjusting the camera’s observation parameters. Furthermore, observing the target in real time with an active camera is an effective method for obtaining the target’s motion information state. The purpose of this paper is to propose an approach based on the scene’s cell division for precisely positioning the target and tracking and observing it in real time using the active camera. As shown in Figure 2, suppose the camera is on the ZOY plane in the world coordinate system, and the target is on the XOY plane. The target area is partitioned into continuous cell areas using predefined rules, and each cell area’s vertices are calibrated.

3.2. Target Localization on an Image

This research employs YOLOv4 to complete the detection and recognition of ship targets in the image. Alexey Bochkovskiy’s team proposed YOLOv4 in 2020. This algorithm is based on the YOLO series [34,35,36,37] and can accurately and rapidly detect and identify targets in images. On the Tesla V100, the YOLOv4 algorithm detects objects in real time at a rate of 65 fps and with an accuracy of 43.5% AP.

The prediction method of the YOLO series algorithm is shown in Figure 3. The recognition result of YOLOv4 is shown in Figure 4. The neural network predicts five bounding boxes on each unit of the feature map, and each bounding box indicates five values, which are $t_x,t_y,t_w,t_h,t_o$. The first four values are coordinates and $b_o$ is the confidence level. The length and width of the bounding box corresponding to the cell are $(p_w,p_h)$, respectively. The distance between the cell and the upper left corner of the image is $(c_x,c_y)$. Then, the predicted value is

$$b_x=\sigma (t_x)+c_x$$

(1)

$$b_y=\sigma (t_y)+c_y$$

(2)

$$b_w=p_w{e}^{t_w}$$

(3)

$$b_h=p_h{e}^{t_h}$$

(4)

$$\mathrm{Pr}\left(\mathrm{object}\right)\ast \mathrm{IOU}(b,\mathrm{object})=s\left(t_o\right)$$

(5)

In the text, (Target: $(b_x,b_y)$) is used to represent the target in the image; “Jeep” and “cross” are the target types; and $(b_x,b_y)$ are the coordinates of the center point of the target bounding box.

We detected the coordinates of the target in the image and the coordinates of all cell vertices with the YOLOv4 algorithm, and then compared the pixel distances from the image’s principal point to all vertices in the image. The distance formula is:

$$S=\sqrt{{(X_O-X_1)}^2+{(Y_O-Y_1)}^2}$$

(6)

We picked out the four vertices closest to the center of the image and considered these four vertices to be the four vertices of a cell. The world coordinates of the principal point of the image in the scene can be calculated with the PTH model. Because the world coordinates of the vertices of each cell are manually calibrated when the scene cell is divided, it is easy to calculate in which scene cell the principal point of the image is located. Finally, the proposed matching method matches the vertices of the image cell with the vertices of the scene cell. For different channels (horizontal or vertical), different matching methods are adopted.

For the horizontal channel, the method used mainly includes two steps. First, we calculated the distance between the vertices of the cell and the vertex of the lower left corner of the image to identify the vertices A1 and A3. Then, we calculated the abscissa of the remaining two vertices. The vertices A2 and A4 can be identified as well. The specific identification method is shown in Figure 5.

For the vertical channel, the method adopted is shown in Figure 6:

When the principal point of the camera is at the starting point, it can be directly determined that the target is in the first cell After the camera moves for the first time, the cell recognition method shown in Figure 6 can be adopted. Figure 6 shows the method to distinguish the cell’s vertices in the vertical channel. Under different shooting angles, the attitude of the cell is different, but its direction is generally as shown in Figure 6. The abscissa value of vertex A2 is the largest, the abscissa value of vertex A4 is the smallest, the ordinate of vertex A1 is the largest, and the ordinate of vertex A3 is the smallest. The four vertices can be distinguished in this way.

3.3. From Image Coordinates to World Coordinates

We set the plane where the target object is located as the XOY plane of the world coordinate system and set the z-axis as the straight line passing through point O and perpendicular to the XOY plane. MN is the image of the target in the camera, as shown in Figure 2. The relationship between the real target and its image appeared as:

$$Z_c\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=K\left[\begin{array}{cc}R& T\\ 0^T& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]$$

(7)

Formula (7) is a classic formula for converting the world coordinate system to the image coordinate system. $Z_c$ is the depth of the point $p(u,v)$ in the camera coordinates, $K$ is the internal parameter matrix of the camera, and R and T are the rotation and translation matrices of the camera coordinate system to the world coordinate system, respectively. When the target is running on the ground, the ground is set to the plane XOY, and the coordinates of the object on the ground in the world coordinate system are set to $(X_w,Y_w,0)$. Then, Formula (7) can also be expressed as:

$$Z_c\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=K\left[\begin{array}{cc}R& T\\ 0^T& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ 0\\ 1\end{array}\right]$$

(8)

Given M as a 3 ∗ 3 matrix, Formula (8) can be represented as follows:

$$s{\left[\begin{array}{ccc}u& v& 1\end{array}\right]}^T=M{\left[\begin{array}{ccc}{X}_w& Y_w& 1\end{array}\right]}^T$$

(9)

where M is the homography matrix and the M matrix has eight degrees of freedom. Therefore, at least four pairs of feature points are required to solve M. Set the expression of the M matrix as follows:

$$M=\left[\begin{array}{ccc}{m}_{11}& m_{12}& m_{13}\\ m_{21}& m_{22}& m_{23}\\ m_{31}& m_{32}& m_{33}\end{array}\right]$$

(10)

From Formulas (9) and (10), we can obtain:

$$\{\begin{array}{c}u=\frac{\frac{m_{11}}{m_{33}}{X}_w+\frac{m_{11}}{m_{33}}{Y}_w+\frac{m_{11}}{m_{33}}}{\frac{m_{11}}{m_{33}}{X}_w+\frac{m_{31}}{m_{33}}{Y}_w+1}\\ v=\frac{\frac{m_{21}}{m_{33}}{X}_w+\frac{m_{22}}{m_{33}}{Y}_w+\frac{m_{23}}{m_{33}}}{\frac{m_{11}}{m_{33}}{X}_w+\frac{m_{31}}{m_{33}}{Y}_w+1}\end{array}$$

(11)

Given the M′ matrix as follows:

$$M^{\prime }={\left[\begin{array}{cccccccc}\frac{m_{11}}{m_{33}}& \frac{m_{12}}{m_{33}}& \frac{m_{13}}{m_{33}}& \frac{m_{21}}{m_{33}}& \frac{m_{22}}{m_{33}}& \frac{m_{23}}{m_{33}}& \frac{m_{31}}{m_{33}}& \frac{m_{32}}{m_{33}}\end{array}\right]}^T$$

(12)

Then, Formula (12) can be expressed as:

$$\left[\begin{array}{c}\begin{array}{cccccccc}{X}_w& Y_w& 1& 0& 0& 0& -u{X}_w& -u{X}_w\end{array}\\ \begin{array}{cccccccc}0& 0& 0& X_w& Y_w& 1& -v{X}_w& -v{Y}_w\end{array}\end{array}\right]M^{\prime }=\left[\begin{array}{c}u\\ v\end{array}\right]$$

(13)

Formula (13) can be regarded as $P{M}^{\prime }=t$, which can be solved by the least square method:

$$M^{\prime }={(P^{T}P)}^{-1}{P}^{T}t$$

(14)

Furthermore, the homography matrix can be obtained without solving the camera’s internal and external parameters. $M$ can be obtained by $M^{\prime }$, Formulas (10) and (12).

As shown in Figure 7, in the camera’s perspective, the image of the red target on the XOY plane is ABCD, but in actual situations, the projection of the target on the XOY plane is ABFG. As a result, the quadrilateral ABCD and the quadrilateral ABFG are quite different. When the YOLOv4 algorithm is used to identify the target, the bounding box of the identified target is shown in Figure 7. The center point of the green bounding box representing the target position deviates further from the target’s true position. Therefore, this paper proposes a position compensation method. The error value of the target at that point is subtracted to obtain the precise position of the target. The specific method is shown in Figure 8.

Two projection models were constructed to calculate the target position. The first projection model is the position model, and the second is the error model. First, the initial position of the target in the world coordinate system is obtained through the position model. Then, the error of the corresponding point of the target in the world coordinate system is obtained through the error model. The position model and error model are shown in Formulas (15) and (16):

$$S_{XY}{\left[\begin{array}{ccc}X& Y& 1\end{array}\right]}^T=M{\left[\begin{array}{ccc}u& v& 1\end{array}\right]}^T$$

(15)

$$S_{Error}{\left[\begin{array}{ccc}E& F& 1\end{array}\right]}^T=M_{Error}{\left[\begin{array}{ccc}u& v& 1\end{array}\right]}^T$$

(16)

From Formulas (15) and (16), the initial coordinates of the target can be solved as $[X,Y,0]$. The coordinate error of the target on the image at point $[X,Y,0]$ is $[E,F,0]$. Finally, the precise world coordinates of the target can be derived:

$${\left[\begin{array}{ccc}x& y& 0\end{array}\right]}^T={\left[\begin{array}{ccc}X& Y& 0\end{array}\right]}^T-{\left[\begin{array}{ccc}E& F& 0\end{array}\right]}^T$$

(17)

3.4. Converts XYZ Coordinates to PTH Coordinates

As shown in Figure 9, the PTZ camera is regarded as a point, and the PTZ camera is located at point O. Set point O as the origin of the world coordinate system, and the coordinates of point A are $[x,y,z]$. The ellipse in Figure 9 is on the ground in the real scene, and the vertices of the cell and the WRC-Jeep in the scene are regarded as points on the ground. According to the coordinate characteristics, it is easy to derive the following formula:

$$\{\begin{array}{l}x=h\cot {\theta }_1·\cos {\theta }_2\\ y=h\cot {\theta }_1·sin{\theta }_2\\ z=h\end{array}$$

(18)

From the above formula,

$$\{\begin{array}{c}{\theta }_1=acr\cot (\sqrt{\frac{x^2+y^2}{z^2}})\ast 180/\pi \\ h=z\end{array}$$

(19)

$${\theta }_2=\{\begin{array}{c}\begin{array}{ccc}\arccos (\frac{x}{\sqrt{x^2+y^2}})\ast 180/\pi & ,& (x>0,y>0)\end{array}\\ \begin{array}{ccc}180-\arccos (\frac{x}{\sqrt{x^2+y^2}})\ast 180/\pi & ,& (x<0,y>0)\end{array}\\ \begin{array}{ccc}180+\arccos (\frac{x}{\sqrt{x^2+y^2}})\ast 180/\pi & ,& (x<0,y<0)\end{array}\\ \begin{array}{ccc}360-\arccos (\frac{x}{\sqrt{x^2+y^2}})\ast 180/\pi & ,& (x>0,y<0)\end{array}\end{array}$$

(20)

According to Formulas (19) and (20), the unique ${\theta }_1$,${\theta }_2$ and H corresponding to point A can be calculated. Therefore, we call the model composed of ${\theta }_1$,${\theta }_2$ and H the PTH model.

4. Experimental Methods

4.1. Experimental Platform and Scene

The proposed method was tested on 50 cm ∗ 250 cm horizontal and vertical channels. The experimental equipment used in the experiment is shown in Figure 10. It mainly includes a PTZ camera. This active camera is produced by Hikvision Company. Its model is DS-2DF8231IW-A, the focal length range of this camera is 6 mm to 186 mm, it has two degrees of freedom of rotation, and the pan and tilt angles are between 0~90 degrees and 0~360 degrees, as shown in Figure 10. In this experiment, the resolution of the captured image is set to 1920 × 1080 (pixels). Another experimental device used in this experiment is a wireless remote-control car, as shown in Figure 10b. The length, width and height of the car are 22.5 cm, 14 cm and 13.5 cm, respectively.

The experimental scene was modeled, and the world coordinate system was constructed (Figure 11a). We divided the operating channel of the WRC-Jeep into a horizontal channel (Figure 11c) and a vertical channel (Figure 11b) and divided each channel into five equally sized cells. The size of each cell was 50 cm ∗ 50 cm, and the vertices of each cell were calibrated. In addition, we calibrated the world coordinates of each vertex.

Before the experiment, the camera needed to be calibrated. The results of camera calibration are as follows:

$$M=\left[\begin{array}{ccc}2807.668433& 0& 963.0151539\\ 0& 2810.606781& 557.4054265\\ 0& 0& 1\end{array}\right]$$

(21)

From the internal parameter matrix, the coordinates of the principal point of the camera can be obtained as: O′ = (963,557).

4.2. Training and Testing of Video Sequences with YOLOv4

This study uses the YOLOv4 algorithm to train and test the sample images. The targets used in the experiment are divided into two categories: “Jeep” and “cross”. The results of training and testing are shown in

Table 1. Training and testing result of YOLOv4.

Target Type	Training Set	Validation Set	Test Set	Accuracy (%)	TPR (%)	FPR (%)
Jeep	4000	800	1200	94.6	96.2	7.1
Cross	4000	800	1200	93.7	95.3	8.2

. The size of the sample picture selected in the experiment is 1920 (pixel) ∗ 1080 (pixel). A good recognition was achieved by applying YOLOv4 for training and testing (Table 1). A total of 6000 pictures were used for testing.

4.3. Cell-Based Scene Division and Calibration

Before the experiment, we first built the world coordinate system in the scene, then divided the experimental scene into cells, and calibrated each vertex of each cell to calibrate the world coordinates of each vertex. At the same time, it was also necessary to calibrate the error of each vertex. That is, when the target center was located at the vertex of the cell from the perspective of the camera, we measured the position error of the WRC-Jeep in the real scene (including the error in the x-axis direction and the error in the y-axis direction) and calibrated all vertices with this method. The calibration results of the horizontal and vertical channels are in

Table 2. Coordinates and errors of cell vertex on horizontal channel.

Vertex Coordinates	Cell_1	Cell_2	Cell_3	Cell_4	Cell_5
XYZ	(50.0, 195, 0)	(100, 195, 0)	(150, 195, 0)	(200, 195, 0)	(250, 195, 0)
	(100, 195, 0)	(150, 195, 0)	(200, 195, 0)	(250, 195, 0)	(300, 195, 0)
	(50.0, 245, 0)	(100, 245, 0)	(150, 245, 0)	(200, 245, 0)	(250, 245, 0)
	(100, 245, 0)	(150, 245, 0)	(200, 245, 0)	(250, 245, 0)	(300, 245, 0)
Errors	(−1.6, 4.2, 0)	(−1.21, 4.01, 0)	(−0.70, 3.80, 0)	(0, 3.47, 0)	(0.97, 3.03, 0)
	(−1.21, 4.01, 0)	(−0.7, 3.80, 0)	(0, 3.47, 0)	(0.97, 3.03, 0)	(2.40, 2.40, 0)
	(−1.80, 5.2, 0)	(−1.37, 5.15, 0)	(−0.76, 5.06, 0)	(0.11, 4.93, 0)	(1.43, 4.74, 0)
	(−1.37, 5.15, 0)	(−0.76, 5.06, 0)	(0.11, 4.93, 0)	(1.43, 4.74, 0)	(3.70, 4.40, 0)

and

Table 3. Coordinates and errors of cell vertex on vertical channel.

Vertex Coordinates	Cell_1	Cell_2	Cell_3	Cell_4	Cell_5
XYZ	(50, −5, 0)	(50, 45, 0)	(50, 95, 0)	(50, 145, 0)	(50, 195, 0)
	(100, −5, 0)	(100, 45, 0)	(100, 95, 0)	(100, 145, 0)	(100, 195, 0)
	(50, 45, 0)	(50, 95, 0)	(50, 145, 0)	(50, 195, 0)	(50, 245, 0)
	(100, 45, 0)	(100, 95, 0)	(100, 145, 0)	(100, 195, 0)	(100, 245, 0)
Errors	(−1.80, −1.10, 0)	(−1.90, −0.16, 0)	(−1.99, 0.73, 0)	(−2.07, 1.57, 0)	(−2.14, 2.37, 0)
	(−0.80, −0.80, 0)	(−0.94, 0.25, 0)	(−1.05, 1.24, 0)	(−1.17, 2.17, 0)	(−1.29, 3.01, 0)
	(−1.90, −0.16, 0)	(−1.99, 0.73, 0)	(−2.07, 1.57, 0)	(−2.14, 2.37, 0)	(−2.2, 3.1, 0)
	(−0.94, 0.25, 0)	(−1.05, 1.24, 0)	(−1.17, 2.17, 0)	(−1.29, 3.01, 0)	(−1.4, 3.8, 0)

. XYZ represents the coordinates of each vertex, and errors represent the coordinate error of each vertex in the camera’s perspective.

4.4. Experimental Result

Tests and verifications were carried out on the horizontal channel and the vertical channel to verify the effectiveness of the proposed method.

4.4.1. Horizontal Channel

We used YOLOv4 to detect the collected images. Figure 12 is a preliminary identification result.

The size of the image is 1920 (pixel) ∗ 1080 (pixel), so the coordinates of the image’s principal point are (963, 557). YOLOv4 detected the coordinates of each vertex (Figure 12). Then, the cell where the image principal point is located in the world coordinate system can be calculated by using the proposed cell recognition method. Then, the world coordinates and error coordinates of each vertex of the cell where the image principal point is located can be determined (Table 2 or Table 3). The homography matrix can be obtained according to the PnP algorithm. Then, the initial world and error coordinates of the WRC-Jeep can be obtained according to the image coordinates of the WRC-Jeep. The fourth column of

Table 4. Calculation results of target’s coordinates of horizontal channel.

No	${\mathit{S}}_{\mathit{X}\mathit{Y}\mathit{Z}}$	${\mathit{H}}_{\mathit{X}\mathit{Y}\mathit{Z}}$	$\mathit{X}\mathit{Y}\mathit{Z}$
1	1.35626763	$\left[\begin{array}{ccc}1.52245695\mathrm{e}-01& 1.40150273\mathrm{e}-01& -1.58787248\mathrm{e}+02\\ 9.05597001\mathrm{e}-02& -9.09602912\mathrm{e}-02& 2.55500386\mathrm{e}+02\\ 3.81641508\mathrm{e}-05& 5.23695156\mathrm{e}-04& 1.00000000\mathrm{e}+00\end{array}\right]$	$\left[\begin{array}{c}72.92019073\\ 224.66570917\\ 0\end{array}\right]$
3	1.11470419	$\left[\begin{array}{ccc}1.21844121\mathrm{e}-01& 8.32514525\mathrm{e}-02& -5.55683535\mathrm{e}+01\\ 3.18156877\mathrm{e}-02& -1.07568140\mathrm{e}-01& 2.76386760\mathrm{e}+02\\ -4.83038187\mathrm{e}-05& 2.98241113\mathrm{e}-04& 1.00000000\mathrm{e}+00\end{array}\right]$	$\left[\begin{array}{c}114.84426517\\ 225.29453574\\ 0\end{array}\right]$
5	1.16552505	$\left[\begin{array}{ccc}1.35185319\mathrm{e}-01& 7.44862982\mathrm{e}-02& -8.53569138\mathrm{e}+00\\ 1.50392413\mathrm{e}-02& -1.08855137\mathrm{e}-01& 3.05124403\mathrm{e}+02\\ -1.88363112\mathrm{e}-05& 3.44206748\mathrm{e}-04& 1.00000000\mathrm{e}+00\end{array}\right]$	$\left[\begin{array}{c}159.02335626\\ 225.72282124\\ 0\end{array}\right]$
7	1.22398391	$\left[\begin{array}{ccc}1.48762446\mathrm{e}-01& 6.83666266\mathrm{e}-02& 4.10077447\mathrm{e}+01\\ -6.60183306\mathrm{e}-03& -1.02742062\mathrm{e}-01& 3.37429622\mathrm{e}+02\\ 8.62145227\mathrm{e}-07& 4.19174940\mathrm{e}-04& 1.00000000\mathrm{e}+00\end{array}\right]$	$\left[\begin{array}{c}201.7738758\\ 224.87612171\\ 0\end{array}\right]$

is the initial world coordinates of the WRC-Jeep, and the fourth column of

Table 5. Calculation results of error coordinates on horizontal channel.

No	${\mathit{S}}_{\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}$	${\mathit{H}}_{\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}$	Errors
1	0.84984742	$\left[\begin{array}{ccc}1.22102100\mathrm{e}-03& 1.45378856\mathrm{e}-04& -2.78884777\mathrm{e}+00\\ -1.11324039\mathrm{e}-03& -3.23401339\mathrm{e}-04& 5.46492558\mathrm{e}+00\\ -4.26978804\mathrm{e}-04& 5.65970168\mathrm{e}-04& 1.00000000\mathrm{e}+00\end{array}\right]$	$\left[\begin{array}{c}-1.53725194\\ 4.70604621\\ 0\end{array}\right]$
3	0.91452474	$\left[\begin{array}{ccc}1.52877053\mathrm{e}-03& 2.33935318\mathrm{e}-06& -2.76170417\mathrm{e}+00\\ -1.48781901\mathrm{e}-03& -2.52296927\mathrm{e}-04& 6.02211248\mathrm{e}+00\\ -4.42526542\mathrm{e}-04& 7.24654529\mathrm{e}-04& 1.00000000\mathrm{e}+00\end{array}\right]$	$\left[\begin{array}{c}-1.1461222\\ 4.60671534\\ 0\end{array}\right]$
5	0.91889583	$\left[\begin{array}{ccc}1.79981944\mathrm{e}-03& -1.64965938\mathrm{e}-04& -2.51493139\mathrm{e}+00\\ -2.10417615\mathrm{e}-03& -3.49829535\mathrm{e}-04& 6.69456194\mathrm{e}+00\\ -4.61714226\mathrm{e}-04& 8.15730153\mathrm{e}-04& 1.00000000\mathrm{e}+00\end{array}\right]$	$\left[\begin{array}{c}-0.61129109\\ 4.47968578\\ 0\end{array}\right]$
7	1.12154179	$\left[\begin{array}{ccc}2.50793394\mathrm{e}-03& -6.01379026\mathrm{e}-04& -2.43508560\mathrm{e}+00\\ -3.03836039\mathrm{e}-03& 1.19343107\mathrm{e}-04& 8.16718248\mathrm{e}+00\\ -4.98552527\mathrm{e}-04& 1.29678885\mathrm{e}-03& 1.00000000\mathrm{e}+00\end{array}\right]$	$\left[\begin{array}{c}0.09275218\\ 4.25034734\\ 0\end{array}\right]$

shows the error coordinates of the WRC-Jeep; the accurate world coordinates of the target can be solved.

Figure 12 and Table 4 and Table 5 show the test results of some frames. The complete trajectory and error of the WRC-Jeep in the horizontal channel are listed; the P value, T value and its error when using the PTZ camera to track and observe the target are listed as well. Finally, the specific results are shown in Figure 13 and Figure 14.

4.4.2. Vertical Channel

We used the same method as the horizontal channel for testing. Figure 15 and Figure 16 show the results of the test under the vertical channel. Figure 15a shows the trajectory of the WRC-Jeep. Figure 15b shows the trajectory error of the WRC-Jeep detected by three different methods. Figure 15 shows the P and T values of the PTZ camera and shows the error between these P and T values with the real P and T values.

According to the experimental results of the horizontal channel and vertical channel, the proposed method has better performance in tracking trajectory and tracking trajectory error. In addition, the value of the observation angle is closer to the ground truth of the observation angle, and the error of the observation angle is smaller than that of the previous method.

Finally, the recognition of the cell vertices is masked to display the position of the WRC-Jeep on the collected images. Then, the coordinates of the WRC-Jeep are displayed after the name of the WRC-Jeep (Figure 17).

5. Conclusions

This paper proposes a target localization and tracking method based on cells and an active camera. In this method, the experimental scene is divided into multiple continuous cells and the vertices of each cell are calibrated. The cell vertices and targets in the experimental scene are detected and recognized by the YOLOv4 algorithm. When the target was running in the experiment scene, we detected which image cell the image principal point was located in, and which cell of the real scene the projection point of the image principal point was located in. Then, we calculated the accurate world coordinates of the target at the current time using the projection model and converted the accurate world coordinates of the target into P, T, and H values using the PTH model for the purpose of tracking and observing the target in the experimental scene with the PTZ camera. This method can accurately locate the target position in the real scene and track the target quickly. The system proposed in this paper has the characteristics of robustness, accuracy, and easy installation.

The method proposed in this paper still has great room for improvement in the future. The proposed method is only implemented and verified in a small space and has good results. When the channel is long or the external conditions are complex, its effect needs to be examined. In addition, when the target runs in a fixed direction in the channel, it will have a better effect; when the target changes its travel attitude or direction in the channel, the target positioning error will increase exponentially.