Video Satellite Staring Control of Ground Targets Based on Visual Velocity Estimation and Uncalibrated Cameras

Abstract

Compared to traditional remote sensing technology, video satellites have unique advantages such as real-time continuous imaging and the ability to independently complete staring observation. To achieve effective staring control, the satellite needs to perform attitude maneuvers to ensure that the target’s projection stays within the camera’s visual field and gradually reaches the desired position. The generation of image-based control instructions relies on the calculation of projection coordinates and their rate of change (i.e., visual velocity) of the projection point on the camera’s image plane. However, the visual velocity is usually difficult to obtain directly. Traditional calculation methods of visual velocity using time differentials are limited by video frame rates and the computing power of onboard processors, and is greatly affected by measurement noise, resulting in decreased control accuracy and a higher consumption of maneuvering energy. In order to address the shortcomings of traditional calculations of visual speed by time difference methods, this paper proposes a control method based on the estimation of visual velocity, which achieves real-time calculation of the target’s visual speed through adaptive estimation; then, the stability of the closed-loop system is rigorously demonstrated. Finally, through simulation comparison with the traditional differential method, the results show that the proposed method has an improvement in attitude accuracy for about 74% and a reduction in energy consumption by about 77%.

1. Introduction

Video satellites have the characteristics of agile attitude maneuvering, real-time continuous video imaging, and the ability to independently complete tracking and observation [1,2,3]. They have unique advantages compared with traditional remote sensing satellites [4]. They can achieve continuous staring observation of ground or space targets, and have broad application prospects in resource exploration, Earth observation, and other fields. Due to the above features, many countries and agencies have attached a greater importance to the research of video satellites, and have successively launched a batch of them in recent years, such as the Skybox [5], LAPAN-TUBSA [6], SkySat [7], Tiantuo-2 [8], and Jilin-1 [9,10] satellites, etc., as a supplement to traditional remote sensing observation methods or ground-based observation methods.

The staring control of video satellites usually requires the camera’s optical axis to point towards the direction of the target, which can make the target projection stay at the center position of the camera’s imaging plane and achieve ideal observation results. It can also avoid the target from leaving the camera’s field of view by leaving a certain distance to the edge of the image plane, ensuring the continuity of the tracking observation. Due to the orbital motion of the video satellite, displacement between the target and the satellite exists. In order to achieve tracking and observation, it is necessary to design attitude controllers for the video satellite.

Early research on video satellite tracking and control required the prior geographic position information of ground targets as the input of the attitude controller in order to calculate attitude errors and then implement staring control [1,11,12,13,14]. This type of method is a location-based tracking control that does not involve the image information obtained by the camera, so the prior position information of the target is required. This means that it cannot be applied to the observation of non-cooperative targets, due to the lack of prior information. Therefore, some scholars have started to conduct research on the staring control of video satellites based on visual image feedback [11,15,16,17,18,19,20,21], which can be applied to the observation of both cooperative and non-cooperative targets.

The methods used in the above studies all assume that the parameters of the onboard camera system are precisely calibrated. However, the calibration of cameras in orbit is difficult and time-consuming, making it difficult to meet the rapid response requirements of increasingly complex space-based observation tasks. At the same time, during the long duration of orbit motion, the camera is affected by factors such as heat and vibration, and its optical properties may also deviate to the nominal value. Adopting a method under the assumption of precise camera calibrations will affect the tracking and control accuracy. Currently, two main methodologies are being utilized to resolve the issue. One is the model revision method, which compensates for the parameter deviation caused by some specific factors (thermal deformation [22], geometric deformation of photosensitive elements [23], etc.) to a certain extent, but is difficult to meet the demand for fast response. The other method compensates for the impact of parameter deviations using control algorithms. At present, this method has been studied (to some extent) in the fields of ground robots [24,25,26,27,28,29,30,31,32] and drones [33,34,35,36,37,38]. However, research on the tracking and control of targets by video satellites with uncalibrated in-orbit cameras is still relatively scarce.

The image-based control method requires establishing the relationship between the relative position of the target with the video satellite and its projection point coordinates on the camera’s imaging plane. The control law is calculated using the deviation between the present projection point and the expected point. Then, under the control maneuver, the attitude of the video satellite is changed, gradually moving the target projection point to the expected position. Therefore, the generation of control instructions depends on the calculation of the change rate of projection point coordinates; that is, the visual speed information. However, in reality, visual speed is difficult to measure directly. In order to calculate the visual speed and use it as an input for the controller, it can be indirectly obtained using traditional methods, by calculating the time differential of the pixel coordinates; that is, the differential calculated value of the visual velocity is obtained by dividing the variation in the target’s projected coordinates at adjacent moments by the time interval. The differential operation accuracy is directly affected by the differences in the time intervals. The smaller the time interval, the higher the visual velocity accuracy obtained. However, the time interval is first limited by the video frame rate of the onboard cameras; for example, the video frame rate of Tiantuo-2 and Jilin-1 is 25 frames per second, while the frame rate of the Surrey V-1C video satellite can reach 100 frames per second. Therefore, even with the same control law, different control effects could occur on video satellites with different frame rate shooting capabilities. Moreover, differential operation is also influenced by algorithms and hardware, with significant differences in image processing speeds between different algorithms and hardware which limit the rapid response capability of video satellites. In addition, the measured pixel coordinates inevitably contain noise effects, and the time difference operation will amplify the impact of noise, which can further affect the control accuracy. Therefore, the visual speed obtained with these differences is not only affected by a lack of smoothness, but is also susceptible to noise, which can reduce control accuracy and stability. Thus, it is necessary to design corresponding adaptive visual tracking control methods for situations where visual velocity is treated as an unknown variable without using differential methods. However, there is still relatively little research on how to obtain the visual velocity when it is difficult to acquire.

This manuscript comprehensively considers the tracking control of video satellites under the conditions of uncalibrated cameras and unknown visual velocity. First, we redesign the control law proposed in Ref. [39] by constructing an adaptive law to estimate the visual speed of the target which does not rely on differential calculations as in the reference. Using real-time camera parameter estimation, the reference attitude trajectory, parameter update law, and tracking controller are all calculated using visual speed estimation values. Simulations demonstrate improved noise robustness and smoother tracking compared to the results in the original reference.

The remaining part of the paper proceeds as follows. In Section 2, the physical models of this article, including the video satellite motion model and camera projection model, are established. In Section 3, we describe the model for estimating visual velocity, redesign the control law, and rigorously prove the stability of the closed-loop system. In Section 4, we conduct comparisons and analyses with the differential calculation of visual velocity through simulation, and we end with some conclusions and open problems in Section 5.

2. Problem Formulation

This section establishes the mathematical models that were used. In order to simplify the article, some formulas have been derived from existing models in Ref. [39], and some of the detailed derivation processes can be found in the original literature. To improve readability, some key variables used in this section are summarized in

Table 1. Variables and their definitions.

Variable	Definition
${{}^{A}R}_{ab}$	Position vector from point a to point b expressed in frame A
${}_A{}^{B}M\in {ℝ}^{3\times 3}$	Euler transform matrix from frame A to frame B
$\Pi \in {ℝ}^{3\times 4}$	Camera intrinsic projection matrix
$T_h\in {ℝ}^{4\times 4}$	Homogeneous transform matrix from ECI to the satellite’s body frame
$T\in {ℝ}^{4\times 4}$	Homogeneous transform matrix from the satellite’s body frame to the camera frame
$N\in {ℝ}^{3\times 4}$	Camera projection matrix

.

2.1. Kinematics and Dynamics of Video Satellite

The kinematic and dynamic equations describe the motion characteristics of the video satellite. Under the assumption of a rigid body, the kinematic and dynamic equations of the video satellite are given by

$$\left[\begin{array}{c}{q}_0\\ {\dot{q}}_v\end{array}\right]={\displaystyle \frac{1}{2}}\left[\begin{array}{c}-q_v^T\\ q_0{I}_{3\times 3}+sk\left(q_v\right)\end{array}\right]\omega$$

(1)

$$J\dot{\omega }=-\omega \times J\omega +U$$

(2)

where J is the moment of inertia of the video satellite, $I_{3\times 3}$ is a 3-order unit matrix, $\omega$ and $q={\left[q_0,q_v^T\right]}^T$ are the angular velocity and attitude quaternion of the video satellite, and $U$ is the attitude control torque. The operator $sk(\cdot )$ is a 3 × 3 skew-symmetric matrix with diagonal elements of 0, with the form of

$$sk\left(q_v\right)=\left[\begin{array}{ccc}0& -q_3& q_2\\ q_3& 0& -q_1\\ -q_2& q_1& 0\end{array}\right]$$

(3)

2.2. Staring Observation Model

When a video satellite performs earth-staring observation tasks, it requires the camera’s optical axis to point towards the ground target, as shown in Figure 1. ${{}^{i}R}_{eb}$ denotes the satellite’s Earth-centered inertial (ECI, $O_e-X_e{Y}_e{Z}_e$) position and ${}_i{}^{b}M$ denotes the transfer matrix from ECI to the satellite’s body frame $O_b-X_b{Y}_b{Z}_b$. Then, the observation model is

$$\left[\begin{array}{c}{{}^{b}R}_{bT}\\ 1\end{array}\right]=T_h\left[\begin{array}{c}{{}^{i}R}_{eT}\\ 1\end{array}\right]$$

(4)

where $T_h=\left[\begin{array}{cc}{}_i{}^{b}M& -{}_i{}^{b}M{{}^{i}R}_{eb}\\ {\mathbf{0}}_{1\times 3}& 1\end{array}\right]\in {ℝ}^{4\times 4}$ is the extended Euler transform matrix from the inertial frame to the satellite’s body frame.

After establishing the staring observation model, it is necessary to convert the position relationship of the target relative to the satellite into a projection relationship. In the following text, the intrinsic and extrinsic projection models of the camera are established separately, and then the projection kinematics model is ultimately derived.

2.3. Camera Intrinsic Model

As shown in Figure 2, the observation target T in the camera frame $O_c-X_c{Y}_c{Z}_c$ is denoted by ${{}^{c}R}_{cT}$ and its projection coordinate in the image plane is $y={\left[u,v\right]}^T$. Then, the projection model of the target is

$$z_c\left[\begin{array}{c}y\\ 1\end{array}\right]=\Pi \left[\begin{array}{c}{{}^{c}R}_{cT}\\ 1\end{array}\right]$$

(5)

where $\Pi =\left[\begin{array}{cccc}{\displaystyle \frac{f}{dx}}& -{\displaystyle \frac{f}{dx}}\cot \phi & u_0& 0\\ 0& {\displaystyle \frac{f}{dy\cdot \sin \phi }}& v_0& 0\\ 0& 0& 1& 0\end{array}\right]$, $\phi$ is the angle between the two axes of the image plane and ${\left[u_0,v_0\right]}^T$ is the intersection point of the optical axis and the image plane.

2.4. Camera Extrinsic Model

As shown in Figure 3, the transformation between ${{}^{c}R}_{cT}$ and ${{}^{b}R}_{bT}$ can be described by

$$\left[\begin{array}{c}{{}^{c}R}_{cT}\\ 1\end{array}\right]=T\left[\begin{array}{c}{{}^{b}R}_{bT}\\ 1\end{array}\right]$$

(6)

where $T=\left[\begin{array}{cc}{}_b{}^{c}M& -{{}^{c}R}_{bc}\\ {\mathbf{0}}_{1\times 3}& 1\end{array}\right]$ is the extended Euler transform matrix between the satellite’s body frame to the onboard camera frame.

2.5. The Projection Kinematics of Staring Imaging

After obtaining the two projection models, the position of the target observed by the camera at the projection point on the imaging plane can be determined. Next, we will separate the projection point’s coordinates to obtain the kinematics of staring imaging.

Combining Equations (5) and (6), we can obtain the relation between the position of the observation target in ECI and its projection coordinates on the image plane; this is

$$z_c\left[\begin{array}{c}y\\ 1\end{array}\right]=N\cdot T_h\cdot \left[\begin{array}{c}{{}^{i}R}_{et}\\ 1\end{array}\right]$$

(7)

where $N\triangleq \Pi \cdot T$ is called the projection matrix, which establishes the relationship between the position of the target in ECI and the coordinates of the camera projection point. Its elements are denoted as $n_{ij}(i=1,2,3;j=1,2,3,4)$.

Expand the matrix $N$ by rows and define it as

$$N\triangleq \left[\begin{array}{c}{n}_1^T\\ n_2^T\\ n_3^T\end{array}\right]\triangleq \left[\begin{array}{c}P\\ n_3^T\end{array}\right]$$

(8)

where $P\in {ℝ}^{2\times 4}$ denotes the first two rows of projection matrix $N$, and $n_3^T\in {ℝ}^{1\times 4}$ is the third row. Then, Equation (7) can be rewritten as

$$\begin{array}{l}y(t)={\displaystyle \frac{1}{z_c(t)}}P\cdot T_h(t)\cdot \left[\begin{array}{c}{{}^{i}R}_{et}(t)\\ 1\end{array}\right]\\ z_c(t)=n_3^T\cdot T_h(t)\cdot \left[\begin{array}{c}{{}^{i}R}_{et}(t)\\ 1\end{array}\right]\end{array}$$

(9)

Denote the first three elements of $n_3^T$ and first three columns of $P$ as $n_{3(3)}^T$ and $P_{(3)}$, respectively. The differentiative of $y(t)$ and $z_c(t)$ can be expressed and simplified as [39]

$$\begin{array}{cc}\hfill \dot{y}(t)& ={\displaystyle \frac{1}{z_c(t)}}\left(P_{(3)}-y(t)n_{3(3)}^T\right)sk\left({}_i{}^{b}M{{}^{i}R}_{eT}(t)-{}_i{}^{b}M{R}_{eb}(t)\right)\omega (t)+\hfill \\ & {\displaystyle \frac{1}{z_c(t)}}\left(P_{(3)}-y(t)n_{3(3)}^T\right){}_i{}^{b}M\left({{}^{i}V}_{eT}(t)-{{}^{i}V}_{eb}(t)\right)\hfill \\ & \triangleq {\displaystyle \frac{1}{z_c(t)}}\left[A(t)\omega (t)+A_v(t)\right]\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill {\dot{z}}_c(t)& =n_{3(3)}^{T}sk\left({}_i{}^{b}M{{}^{i}R}_{eT}(t)-{}_i{}^{b}M{R}_{eb}(t)\right)\omega (t)+n_{3(3)}^T{}_i{}^{b}M\left({{}^{i}V}_{eT}(t)-{{}^{i}V}_{eb}(t)\right)\hfill \\ & \triangleq a(t)\omega (t)+a_v(t)\hfill \end{array}$$

(11)

Equations (10) and (11) are called the staring imaging kinematics. $\dot{y}(t)$ is the rate of change in the coordinates of the target on the imaging plane of the onboard camera over time, i.e., the visual velocity.

3. Design of Adaptive Control Law

3.1. Estimation of Visual Velocity

Visual velocity is a crucial input for the controller. When it is unknown, real-time estimation is required. In the manuscript, we define $\widehat{a}$ as the estimated value of a. Then, according to Equation (10), the current estimated value of the real-time visual velocity can be expressed as

$$\widehat{\dot{y}}(t)={\displaystyle \frac{1}{{\widehat{z}}_c(t)}}\left[\widehat{A}(t)\widehat{\omega }(t)+{\widehat{A}}_v(t)\right]$$

(12)

Combining Equations (10) and (12), the estimation error of visual velocity is defined as

$$\begin{array}{cc}\hfill \dot{y}(t)-\widehat{\dot{y}}(t)& ={\displaystyle \frac{1}{{\widehat{z}}_c(t)z_c(t)}}\left[\left({\widehat{z}}_c(t)A(t)-z_c(t)\widehat{A}(t)\right)\omega (t)+\left({\widehat{z}}_c(t)A_v(t)-z_c(t){\widehat{A}}_v(t)\right)\right]\hfill \\ & ={\displaystyle \frac{1}{{\widehat{z}}_c(t)z_c(t)}}\{\left[{\widehat{z}}_c(t)\left(A(t)-\widehat{A}(t)\right)+\left({\widehat{z}}_c(t)-z_c(t)\right)\widehat{A}(t)\right]\omega (t)+\hfill \\ & {\widehat{z}}_c(t)\left(A_v(t)-{\widehat{A}}_v(t)\right)+\left({\widehat{z}}_c(t)-z_c(t)\right){\widehat{A}}_v(t)\}\hfill \\ & \triangleq {\displaystyle \frac{1}{{\widehat{z}}_c(t)z_c(t)}}Q(t)\Delta {\theta }_p\hfill \end{array}$$

(13)

where ${\theta }_p={\left[n_{11},n_{12},n_{13},n_{14},n_{21},n_{22},n_{23},n_{24},n_{31},n_{32},n_{34}\right]}^T$ is the estimation of the elements in matrix N and $\Delta {\theta }_p$ is the estimation error, which is the same as the definition in Ref. [39].

Through the intermediate step of the above equation, it can be seen that the estimation error expression of visual velocity contains the parameter “estimation error” ($\Delta {\theta }_p$), so it can be represented by linearization. It should be noted that after extracting $\Delta {\theta }_p$, the remaining expressions still contain parameter estimation values, so $Q(t)$ is related to the parameter estimation results ${\theta }_p$.

3.2. Reference Attitude Trajectory

When designing the controller, the deviation between the current state and the expected state is usually used as its input. Due to the difficulty in directly calculating the expected angular velocity ${\omega }_d$, we indirectly calculated the attitude error by defining a reference attitude trajectory. In Ref. [39], the reference attitude trajectory ${\dot{y}}_r(t)$, reference image tracking error ${\delta }_{\dot{y}}(t)$, and reference angular velocity tracking error ${\omega }_r(t)$ are defined as

$${\dot{y}}_r(t)=-\lambda \Delta y(t)$$

(14)

$${\delta }_{\dot{y}}(t)=\dot{y}(t)-{\dot{y}}_r(t)=\Delta \dot{y}(t)+\lambda \Delta y(t)$$

(15)

$${\delta }_{\omega }(t)=\omega (t)-{\omega }_r(t)$$

(16)

where $\Delta y(t)=y(t)-y_d$ is the image error between the current projected position and the expected position. ${\omega }_r(t)$ is defined as

$${\omega }_r(t)={\widehat{A}}^{+}(t)\left(-{\widehat{z}}_c(t)\cdot \lambda \cdot \Delta y(t)-{\widehat{A}}_v(t)\right)$$

(17)

It can be seen that ${\dot{y}}_r(t)$ is determined solely by $\Delta y(t)$ and is unrelated to the unknown visual velocity $\dot{y}(t)$. Therefore, there is no need to re estimate ${\dot{y}}_r(t)$. However, ${\delta }_{\dot{y}}(t)$ is related to $\dot{y}(t)$, so it is necessary to use the estimated values of visual velocity instead, which leads to

$${\widehat{\delta }}_{\dot{y}}(t)=\widehat{\dot{y}}(t)-{\widehat{\dot{y}}}_r(t)$$

(18)

By combining Equations (12) and (17), the estimated image tracking error ${\widehat{\delta }}_{\dot{y}}(t)$ satisfies

$$\begin{array}{cc}\hfill {\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)& ={\widehat{z}}_c(t)\widehat{\dot{y}}(t)-{\widehat{z}}_c(t){\dot{y}}_r(t)\hfill \\ & =\left(\widehat{A}(t)\omega (t)+{\widehat{A}}_v(t)\right)-\left(\widehat{A}(t){\omega }_r(t)+{\widehat{A}}_v(t)\right)\hfill \\ & =\widehat{A}{\delta }_{\omega }(t)\hfill \end{array}$$

(19)

By taking the derivative of Equation (17), we can obtain the rate of change of the reference angular velocity

$$\begin{array}{cc}\hfill {\dot{\omega }}_r(t)& ={\displaystyle \frac{\mathrm{d}{\widehat{A}}^{+}(t)}{\mathrm{d}t}}{\widehat{z}}_c(t)\cdot {\dot{y}}_r(t)+{\widehat{A}}^{+}(t){\displaystyle \frac{\mathrm{d}{\widehat{z}}_c(t)}{\mathrm{d}t}}{\dot{y}}_r(t)+{\widehat{A}}^{+}(t){\widehat{z}}_c(t){\ddot{y}}_r(t)\hfill \\ & -{\displaystyle \frac{\mathrm{d}{\widehat{A}}^{+}(t)}{\mathrm{d}t}}{\widehat{A}}_v(t)-{\widehat{A}}^{+}(t){\displaystyle \frac{\mathrm{d}{\widehat{A}}_v(t)}{\mathrm{d}t}}\hfill \end{array}$$

(20)

It can be seen that some terms in (20) do not include visual velocity, while the third term that includes visual velocity can be expressed in a linear form (since ${\ddot{y}}_r(t)=-\lambda \dot{y}(t)$). Therefore, the reference angular acceleration can be expressed in a simplified way:

$${\dot{\omega }}_r(t)=R(t){\widehat{z}}_c(t)\dot{y}(t)+{\overline{\dot{\omega }}}_r(t)$$

(21)

where ${\overline{\dot{\omega }}}_r(t)$ is part of the expression of ${\dot{\omega }}_r(t)$, which is independent of visual velocity. The matrix $R(t)$ is defined by linearizing the visual velocity and removing the estimated depth. Due to the unknown $\dot{y}(t)$, the estimated value $\widehat{\dot{y}}(t)$ is substituted into (21) to get

$${\widehat{\dot{\omega }}}_r(t)=R(t){\widehat{z}}_c(t)\widehat{\dot{y}}(t)+{\overline{\dot{\omega }}}_r(t)$$

(22)

Subtracting Equation (21) from Equation (22), the difference between the reference angular acceleration and its estimated value can be obtained by combining them with Equation (13), resulting in

$$\begin{array}{cc}\hfill {\widehat{\dot{\omega }}}_r(t)-{\dot{\omega }}_r(t)& =R(t){\widehat{z}}_c(t)\left(\widehat{\dot{y}}(t)-\dot{y}(t)\right)\hfill \\ & =-{\displaystyle \frac{1}{z_c(t)}}R(t)Q(t)\Delta {\theta }_p\hfill \end{array}$$

(23)

3.3. Design of Control Law

3.3.1. Parameter Definition and Estimation Law

Firstly, we need to calculate the estimation error, including the projection coordinate estimation error and camera parameter estimation error. The estimated projection error is defined in the same way as Ref. [39] with the expression of

$$e(t)={\widehat{z}}_c(t)y(t)-P(t)T_h\left[\begin{array}{c}{{}^{i}R}_{et}(t)\\ 1\end{array}\right]$$

(24)

The parameter update law is designed as

$${\dot{\widehat{\theta }}}_p=-{\Gamma }^{-1}\left\{{\left(JR(t)Q(t)\right)}^T{\delta }_{\omega }(t)+W_p^T{K}_{1}e(t)+K_2{\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}{\Vert {\delta }_{\omega }(t)\Vert }^2\right\}$$

(25)

where $\Gamma$, $K_1$, and $K_2$ are all positive definite matrix.

The key difference from the parameter update law in Ref. [39] is the first term, which is used to simplify the process of closed-loop stability proof. $K_1$ and $K_2$ are selected to make the system stable. The stability proof will be conducted in the following text.

3.3.2. Staring Controller Design

By redesigning the controller in Ref. [39] and using the estimated parameters as input, the designed control law can be obtained as

$$\begin{array}{cc}\hfill U(t)& =\omega (t)\times J\omega (t)+J{\widehat{\dot{\omega }}}_r(t)-{\widehat{A}}^T{K}_3{\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)-K_4{\delta }_{\omega }(t)-\hfill \\ & K_5\Vert {\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}\Vert {\delta }_{\omega }(t)-K_6\Vert \omega (t)\Vert {\delta }_{\omega }(t)\hfill \end{array}$$

(26)

where $K_3$, $K_4$, $K_5$, and $K_6$ are positive definite matrix. Substituting the control law expression into (2), we can obtain

$$\begin{array}{cc}\hfill J\dot{\omega }(t)& =J{\widehat{\dot{\omega }}}_r(t)-{\widehat{A}}^T(t)K_3{\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)-K_4{\delta }_{\omega }(t)-\hfill \\ & K_5\Vert {\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}\Vert {\delta }_{\omega }(t)-K_6\Vert \omega (t)\Vert {\delta }_{\omega }(t)\hfill \end{array}$$

(27)

Combining Equation (23), the following expression yields

$$\begin{array}{cc}\hfill J{\dot{\delta }}_{\omega }(t)& =J\dot{\omega }(t)-J{\dot{\omega }}_r(t)\hfill \\ & =J\left({\widehat{\dot{\omega }}}_r(t)-{\dot{\omega }}_r(t)\right)-{\widehat{A}}^T(t)K_3{\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)-K_4{\delta }_{\omega }(t)-\hfill \\ & K_5\Vert {\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}\Vert {\delta }_{\omega }(t)-K_6\Vert \omega (t)\Vert {\delta }_{\omega }(t)\hfill \\ & =-{\displaystyle \frac{J}{z_c(t)}}R(t)Q(t)\Delta {\theta }_p-{\widehat{A}}^T(t)K_3{\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)-K_4{\delta }_{\omega }(t)-\hfill \\ & K_5\Vert {\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}\Vert {\delta }_{\omega }(t)-K_6\Vert \omega (t)\Vert {\delta }_{\omega }(t)\hfill \end{array}$$

(28)

3.3.3. Stability Analysis

Choosing an appropriate positive-definite Lyapunov function and analyzing its time derivative is a common method for analyzing the stability of the system. Here we construct the following two non-negative functions to conduct stability analysis:

$$V_1={\displaystyle \frac{1}{2}}{z}_c(t){\delta }_{\omega }^T(t)J{\delta }_{\omega }(t)$$

(29)

$$V_2={\displaystyle \frac{1}{2}}\Delta {\theta }_p^T(t)\Gamma \Delta {\theta }_p(t)$$

(30)

Taking the derivative of $V_1$, we can obtain

$$\begin{array}{cc}\hfill {\dot{V}}_1& ={\displaystyle \frac{1}{2}}{\dot{z}}_c(t){\delta }_{\omega }^T(t)J{\delta }_{\omega }(t)+z_c(t){\delta }_{\omega }^T(t)J{\dot{\delta }}_{\omega }(t)\hfill \\ & ={\displaystyle \frac{1}{2}}{\dot{z}}_c(t){\delta }_{\omega }^T(t)J{\delta }_{\omega }(t)+z_c(t){\delta }_{\omega }^T(t)[-{\displaystyle \frac{J}{z_c(t)}}R(t)Q(t)\Delta {\theta }_p-{\widehat{A}}^T(t)K_3{\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)-K_4{\delta }_{\omega }(t)-\hfill \\ & K_5\Vert {\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}\Vert {\delta }_{\omega }(t)-K_6\Vert \omega (t)\Vert {\delta }_{\omega }(t)]\hfill \\ & ={\displaystyle \frac{1}{2}}{\dot{z}}_c(t){\delta }_{\omega }^T(t)J{\delta }_{\omega }(t)-{\delta }_{\omega }^T(t)JR(t)Q(t)\Delta {\theta }_p-z_c(t){\delta }_{\omega }^T(t){\widehat{A}}^T(t)K_3{\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)-z_c(t){\delta }_{\omega }^T(t)K_4{\delta }_{\omega }(t)-\hfill \\ & z_c(t){\delta }_{\omega }^T(t)K_5\Vert {\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}\Vert {\delta }_{\omega }(t)-z_c(t){\delta }_{\omega }^T(t)K_6\Vert \omega (t)\Vert {\delta }_{\omega }(t)\hfill \end{array}$$

(31)

Substituting Equation (19) into the above equation, we can obtain

$$\begin{array}{cc}\hfill {\dot{V}}_1& ={\displaystyle \frac{1}{2}}{\dot{z}}_c(t){\delta }_{\omega }^T(t)J{\delta }_{\omega }(t)-{\delta }_{\omega }^T(t)JR(t)Q(t)\Delta {\theta }_p-z_c(t){\left({\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)\right)}^T{K}_3{\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)-z_c(t){\delta }_{\omega }^T(t)K_4{\delta }_{\omega }(t)\hfill \\ & z_c(t){\delta }_{\omega }^T(t)K_5\Vert {\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}\Vert {\delta }_{\omega }(t)-z_c(t){\delta }_{\omega }^T(t)K_6\Vert \omega (t)\Vert {\delta }_{\omega }(t)\hfill \end{array}$$

(32)

Taking the derivative of $V_2$ and combining it with Equation (25), we can obtain

$$\begin{array}{cc}\hfill {\dot{V}}_2& =\Delta {\theta }_p^T(t)\Gamma \Delta {\dot{\theta }}_p(t)\hfill \\ & =-\Delta {\theta }_p^T(t)\left\{{\left(JR(t)Q(t)\right)}^T{\delta }_{\omega }(t)+W_p^T{K}_{1}e(t)+K_2{\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}{\Vert {\delta }_{\omega }(t)\Vert }^2\right\}\hfill \\ & =-\Delta {\theta }_p^T(t){\left(JR(t)Q(t)\right)}^T{\delta }_{\omega }(t)-e^T{K}_{1}e(t)-\Delta {\theta }_p^T(t)K_2{\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}{\Vert {\delta }_{\omega }(t)\Vert }^2\hfill \end{array}$$

(33)

The Lyapunov function is taken as $V(t)=V_1(t)+V_2(t)$; then, we can obtain

$$\begin{array}{cc}\hfill \dot{V}(t)& ={\dot{V}}_1(t)+{\dot{V}}_2(t)\hfill \\ & =-e^T{K}_{1}e(t)-z_c(t){\left({\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)\right)}^T{K}_3{\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)-\hfill \\ & {\delta }_{\omega }^T(t)\left[z_c(t)K_6\Vert \omega (t)\Vert +z_c(t)K_4-{\displaystyle \frac{1}{2}}\left(a(t)\omega (t)+a_v(t)\right)\right]{\delta }_{\omega }(t)-\hfill \\ & z_c(t){\delta }_{\omega }^T(t)K_5\Vert {\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}\Vert {\delta }_{\omega }(t)-\Delta {\theta }_p^T(t)K_2{\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}{\Vert {\delta }_{\omega }(t)\Vert }^2\hfill \end{array}$$

(34)

Next, we will analyze the derivative of the Lyapunov function to determine the stability of the closed-loop system. According to Equation (11), $a(t)$ and $a_v(t)$ are both bounded functions; therefore, it is assumed that

$$\begin{array}{l}\Vert a(t)\Vert \le \chi \\ \Vert a_v(t)\Vert \le {\chi }_v\end{array}$$

(35)

According to the property that the dot product of vectors is less than or equal to the product of the vector module length, by scaling Equation (34), we can obtain

$$\begin{array}{cc}\hfill \dot{V}(t)& \le -e^T{K}_{1}e(t)-z_c(t){\left({\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)\right)}^T{K}_3{\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)-\hfill \\ & {\delta }_{\omega }^T(t)\left[z_c(t)K_6\Vert \omega (t)\Vert +z_c(t)K_4-{\displaystyle \frac{1}{2}}\left(\chi \Vert \omega (t)\Vert +{\chi }_v\right)\right]{\delta }_{\omega }(t)-\hfill \\ & \left(z_c(t)K_5-\Vert \Delta {\theta }_p^T(t)\Vert K_2\right)\Vert {\displaystyle \frac{\partial U\left({\widehat{\theta }}_p(t)\right)}{\partial {\widehat{\theta }}_p(t)}}\Vert {\Vert {\delta }_{\omega }(t)\Vert }^2\hfill \end{array}$$

(36)

$\min (z_c)$ represents the minimum value of $z_c$, when the following inequality holds

$$\begin{array}{l}{k}_{4\min }\ge {\displaystyle \frac{{\chi }_v}{2\min (z_c)}}\\ k_{5\min }\ge {\displaystyle \frac{k_{2\max }}{\min (z_c)}}\sqrt{{\displaystyle \frac{2V(0)}{{\tau }_{\min }}}}\\ k_{6\min }\ge {\displaystyle \frac{\chi }{2\min (z_c)}}\end{array}$$

(37)

We have

$$\dot{V}(t)\le 0$$

(38)

where $k_{4\min }$, $k_{5\min }$, $k_{6\min }$, and ${\tau }_{\min }$ are the minimum eigenvalues of $K_4$, $K_5$, $K_6$, and $\Gamma$, $k_{2\max }$ is the maximum eigenvalue of $K_2$. Thus, $\dot{V}(t)$ is bounded, and so are $\widehat{\dot{y}}(t)$, $\omega (t)$, ${\omega }_r(t)$, $e(t)$ and ${\widehat{\theta }}_p(t)$. According to Equation (25), the derivative of the estimated parameter values is also bounded. Furthermore, it can be inferred that the derivative of the estimated projection error $\dot{e}(t)$ is also bounded.

According to Equation (22), it can be inferred that ${\widehat{\dot{\omega }}}_r(t)$ is also a bounded function, and so is $U(t)$. Therefore, combining the attitude dynamics of the video satellite, the boundedness of the angular acceleration $\dot{\omega }(t)$ can be obtained. Based on the above analysis, it can be concluded based on Barbalat’s lemma that

$$\underset{t\to \infty }{\lim }\dot{V}(t)=0$$

(39)

that is

$$\begin{array}{c}\underset{t\to \infty }{\lim }{\delta }_{\omega }(t)=\mathbf{0}\\ \underset{t\to \infty }{\lim }{\widehat{z}}_c(t){\widehat{\delta }}_{\dot{y}}(t)=\mathbf{0}\\ \underset{t\to \infty }{\lim }e(t)=\mathbf{0}\end{array}$$

(40)

According to Equation (24), since the parameter estimation values will not all be zero, when $e(t)$ is approaching 0, the depth estimation value cannot be zero, resulting in

$$\underset{t\to \infty }{\lim }{\widehat{\delta }}_{\dot{y}}(t)=\mathbf{0}$$

(41)

Due to the inclusion of $\Delta \dot{y}(t)$ and $\Delta y(t)$ in ${\delta }_{\dot{y}}(t)$, it is not yet possible to determine the convergence of the pixel error. Next, we will further prove that the projection point will converge to the expected position in the image plane. By estimating the pixel coordinates through parameter estimation values, the estimated pixel coordinates can be defined as

$$\widehat{y}(t)={\displaystyle \frac{1}{{\widehat{z}}_c(t)}}\widehat{P}\cdot T_h(t)\cdot \left[\begin{array}{c}{{}^{i}R}_{et}(t)\\ 1\end{array}\right]$$

(42)

When $e(t)$ is approaching 0, we have

$$\underset{t\to \infty }{\lim }{\displaystyle \frac{1}{{\widehat{z}}_c(t)}}\widehat{P}\cdot T_h(t)\cdot \left[\begin{array}{c}{{}^{i}R}_{et}(t)\\ 1\end{array}\right]=\underset{t\to \infty }{\lim }y(t)$$

(43)

Therefore, according to Equations (42) and (43), it can be concluded that

$$\underset{t\to \infty }{\lim }\widehat{y}(t)=\underset{t\to \infty }{\lim }y(t)$$

(44)

$$\underset{t\to \infty }{\lim }\dot{\widehat{y}}(t)=\underset{t\to \infty }{\lim }\dot{y}(t)$$

(45)

According to the differentiation rule

$$\dot{\widehat{y}}(t)=\widehat{\dot{y}}(t)+{\displaystyle \frac{\partial \widehat{y}(t)}{\partial {\theta }_p(t)}}{\dot{\theta }}_p(t)$$

(46)

By substituting Equation (40) obtained from Barbalat’s lemma into the expression of the parameter update law (25), it can be seen that the derivative of the estimated parameter value ${\dot{\theta }}_p(t)$ also tends to zero. Therefore, by substituting it into the above equation, we obtain

$$\underset{t\to \infty }{\lim }\dot{\widehat{y}}(t)=\underset{t\to \infty }{\lim }\widehat{\dot{y}}(t)$$

(47)

According to Equations (45) and (47), we get

$$\underset{t\to \infty }{\lim }\widehat{\dot{y}}(t)=\underset{t\to \infty }{\lim }\dot{y}(t)$$

(48)

The above equation indicates that the estimated visual velocity will eventually approach the actual visual velocity. Therefore, it can be inferred that

$$\underset{t\to \infty }{\lim }{\delta }_{\dot{y}}(t)=\underset{t\to \infty }{\lim }{\widehat{\delta }}_{\dot{y}}(t)=\mathbf{0}$$

(49)

This means that the visual tracking error will eventually converge. Therefore, under the designed adaptive visual tracking controller, the estimated visual velocity gradually approaches the actual value, the projection of the ground target gradually approaches the expected coordinates, and then finally stabilizes at the center of the image.

Based on the above derivation process, it can be seen that the process of proving the convergence of visual velocity estimation values does not rely on the sampling time step. Therefore, this method can effectively estimate the visual velocity under different video frame rates, thereby achieving effective tracking of the target. However, for the differential time calculation method, its calculation accuracy of visual velocity will be directly related to the video frame rate. Therefore, this method has stronger universality and can achieve more accurate visual velocity estimation for satellites with different video sampling capabilities. Next, we will validate this method through simulation comparison.

4. Simulation

4.1. Parameter Setting

In this section, we compare the proposed controller based on visual velocity estimation with the differential method in Ref. [39] for ground target observation.

The initial (12 January 2025 12:30:00 UTC) location of the ground target is shown in

Table 2. Location of the observed ground target.

Longitude/Deg	Latitude/Deg	Height/m
128.271	64.72	0

. The orbit elements of the video satellite are shown in

Table 3. Orbit elements of the video satellite.

a/km	e	i/Deg	Ω/Deg	ω/Deg	f/Deg
6868.14	0	97.2574	290.017	59.3884	54.8163

. The parameters of the onboard camera are given in

Table 4. Camera parameters.

Camera Parameters	Nominal Values	Actual Values
f	1 m	1.1 m
$\left(u_0,v_0\right)$	(376, 291)	(396, 276)
$dx$	8.33 μm	8.43 μm
$dy$	8.33 μm	8.43 μm
${{}^{c}R}_{bc}$	${\left[-0.2682,0.0408,0.0671\right]}^T$ m	${\left[-0.2582,0.0358,0.0771\right]}^T$ m
${}_b{}^{c}M$	$M_{321}\left(-{30}^{\circ },{40}^{\circ },-{20}^{\circ }\right)$	$M_{321}\left(-{29}^{\circ },{39.6}^{\circ },-{18.9}^{\circ }\right)$

, in which $M_{321}$ represents the Euler transformation matrix in the order of 3-2-1. The parameters of the controller designed in this paper are shown in

Table 5. Controller parameters.

Controller Parameters	Values
$K_1$	${10}^{-15}\times I_{2\times 2}$
$K_2$	10−3
$K_3$	$2.5\times {10}^{-15}{I}_{2\times 2}$
$K_4$	$2\times I_{2\times 2}$
$K_5$	$30\times I_{3\times 3}$
$K_6$	$0.8\times I_{3\times 3}$
$\Gamma$	$\mathrm{diag}(8\times {10}^{-4}\times I_{8\times 8},1000\times I_{3\times 3})$
a	1
b	0.001

. The compared controller parameters are consistent with Ref. [39]. The nominal parameters are initially real states of the camera and are used as the initial estimated values. In long term obit motion, the actual parameters deviate from the nominal values. $\left(u_0,v_0\right)$ is the desired projection position of the observed target. The initial attitude motion states of the satellite are shown in

Table 6. Initial attitude motion states of the satellite.

Attitude Quaternion	Angular Velocity/Deg·s−1
${\left[0.4228,0.6600,0.5414,0.3040\right]}^T$	${\left[0.0178,0.0587,0.0167\right]}^T$

. It is assumed that the maximum control torque of the actuator is 0.1 N·m; that is, $\Vert U_i(t)\Vert \le 0.1 \mathrm{N}\cdot \mathrm{m}$ holds for $i=1,2,3$.

The simulation comparative analysis includes two parts. First, the results of the traditional time difference method under different time steps are analyzed with the visual velocity estimation method. Then, we add noise in the pixel coordinates to simulate measurement errors in the image processing process to comparatively analyze the effects of the two methods.

In order to quantitatively analyze the performance of the two methods, we define two indicators that characterize the energy consumption during the control process and the terminal stability, with the expressions of

$$I_{energy}={\displaystyle {\int }_0^T\Vert T_c\Vert }\mathrm{d}t$$

(50)

$$I_{stability}^y=\sqrt{{\displaystyle \frac{1}{t_{stable}}}{\displaystyle {\int }_{T-t_{stable}}^T{\left(y(t)-y_d\right)}^2\mathrm{d}t}}$$

(51)

where $T$ and $t_{stable}$ are the duration of the simulation and stable stage, respectively. It can be seen that $I_{energy}$ can characterize the energy consumption during the control process and $I_{stability}^y$ can indicate the steady-state accuracy of the attitude. In the following simulation, $T$ and $t_{stable}$ are set as 10 s and 5 s respectively.

4.2. Simulation Analysis

4.2.1. Effect of Differential Time Step

Figure 4, Figure 5 and Figure 6 respectively demonstrate the control effect when estimating visual velocity using differential time intervals of 0.1 s, 0.08 s, and 0.05 s. The left picture shows the target projection trajectory in the image plane, and the right image shows the angular velocity variation curve of the video satellite.

By comparing and observing Figure 4, Figure 5 and Figure 6 and the data in

Table 7. Comparison of three indicators with different differential time steps.

Time Step	${\mathit{I}}_{\mathit{e}\mathit{n}\mathit{e}\mathit{r}\mathit{g}\mathit{y}}$	${\mathit{I}}_{\mathit{s}\mathit{t}\mathit{a}\mathit{b}\mathit{i}\mathit{l}\mathit{i}\mathit{t}\mathit{y}}^{\mathit{y}}$
0.1 s	0.1457	6.1482
0.08 s	0.1021	5.6921
0.05 s	0.0923	5.5498

, it can be seen that for the differential control method, the time difference step has an obvious impact on the control effect. As can be seen in Figure 4, Figure 5 and Figure 6, the smaller differential steps result in a higher steady-state control accuracy and lower energy consumption. Therefore, in order to improve the control accuracy and efficiency, it is necessary to increase the frequency of the differential calculation. However, this requires higher video frame rates, more efficient image recognition algorithms, and stronger microcontrollers. Therefore, it is not a feasible approach in practice due to the performance of the onboard camera (as previously specified).

Figure 7 shows the pixel coordinate trajectory and angular velocity curve obtained by the controller designed in this paper. It can be seen that this method also achieves the adjustment of the satellite to the desired attitude. Moreover, the trajectory of the target projection (left figure) and the curve of the satellite’s angular velocity (right figure) are smoother compared with the above results corresponding to three differential time steps. There is almost no oscillation during the control process. After calculation, the indicators for this controller are $I_{energy}=0.0634$ and $I_{stability}^y=1.2574$. From the perspective of attitude control stability, since $I_{stability}^y$ is less than the results of the three differential calculations, it indicates that the deviation between the target projection point and the expected point in the stable stage is smaller. From the perspective of control efficiency, since the $I_{energy}$ is also the minimum value among the four results, it indicates that the method based on visual velocity estimation consumes less energy within the same control time.

4.2.2. Impact of Measurement Noise

The process of extracting the pixel coordinates of a target from an image is accomplished by image processing techniques. The extracted pixel coordinates inevitably contain noise, and there is a certain deviation between the extracted target coordinates and the actual coordinates. For the control method of obtaining visual velocity through the differential method, the differential will also introduce noise into the visual velocity, so the effect of noise on the accuracy will be even more severe.

In this subsection, ±7 pixels of random noise were added to the actual pixel coordinates. The differential time interval was set to be 0.08 s, and the control effects of the two methods were compared under the same conditions.

Figure 8 and Figure 9 show the control effects, and

Table 8. Comparison of three indicators for two controllers under noise condition.

Controller	${\mathit{I}}_{\mathit{e}\mathit{n}\mathit{e}\mathit{r}\mathit{g}\mathit{y}}$	${\mathit{I}}_{\mathit{s}\mathit{t}\mathit{a}\mathit{b}\mathit{i}\mathit{l}\mathit{i}\mathit{t}\mathit{y}}^{\mathit{y}}$
Time differential	0.3968	19.4731
Estimation of visual velocity	0.0894	4.9598

records the two indicators for the two methods. From the two figures, it can be seen that for the control method of obtaining visual velocity using differential methods, both the control torque and angular velocity curve demonstrate significant oscillations. However, for the control method proposed in this paper, the target projection trajectory and control torque are relatively smooth, with almost no oscillation.

This is because the visual velocity term plays a very important role in the control law, and the differential operation amplifies the impact of measurement noise, ultimately reflecting in the oscillation of the control torque. However, for the method designed in this paper, the visual velocity is estimated by camera parameters and is not directly affected by noise, so the image noise has a relatively smaller impact.

Comparing the values of the indicators for the two control methods in Table 8, it can also be seen that compared to the differential method, the method proposed in this paper can effectively overcome the influence of noise and achieve more stable attitude control (with the accuracy increased by about 74.53%) with lower energy consumption (reduced by about 77.47%). In addition, the peak control torque required by the method in this article is smaller than that required by the differential method, so the requirement for the maneuverability of the actuator is also lower.

5. Conclusions

To address the challenges of obtaining visual velocity and mitigating noise effects in traditional differential methods, we propose an adaptive control method based on visual velocity estimation. The estimation of visual velocity is achieved through the camera parameters, which avoids the drawbacks of time differentials. The simulation results indicate that the goal of staring control is achieved by the proposed method. Compared with the traditional differential method, the attitude accuracy is improved by about 74.53% and the energy consumption of the control process is reduced by about 77.47% under the noise condition.

The method proposed in this article is aimed at tracking and observing ground targets, whose motion is predictable due to earth rotation. However, for a large number of non-cooperative spatial targets or ground moving targets, it is difficult to obtain their motion information. Further research is needed to consider how to effectively estimate their visual velocity and track their motion without sufficient prior information.