Inter-Spacecraft Rapid Transfer Alignment Based on Attitude Plus Angular Rate Matching Using Q-Learning Kalman Filter

Abstract

This study focuses on the transfer alignment issue between a master spacecraft and a slave spacecraft for the scenario in which the slave spacecraft is mounted on the master satellite before release and should be ready to depart and perform its space mission independently. The challenge of the transfer alignment is to estimate the attitude and calibration parameters of the gyroscope unit (GU) on the slave spacecraft based on the attitude determination system (ADS) of the master spacecraft. To improve the accuracy and rapidity of the transfer alignment, a novel attitude plus angular rate matching scheme is presented using fused sensor information on the master spacecraft. Accordingly, a fifteen-dimensional state-space model is derived to estimate the spacecraft attitude, the GU bias, scale factor error and misalignment simultaneously. A Q-learning Kalman filter (QKF) is designed to fine tune the process noise covariance matrix related to the calibration parameters, which benefits the state estimation performance. The simulation results show that the presented attitude plus angular rate matching scheme performs better than the traditional attitude matching scheme, and the QKF outperforms the standard Kalman filter (KF) and the adaptive Kalman filter (AKF).

1. Introduction

Recently, there have been many space systems that contain the master spacecraft and the slave spacecraft, such as CubeSats deployed from the space station [1], the lander or rover released from a deep-space probe on the orbit of a planet [2] and the smart impactor launched from a mother flyby spacecraft [3]. Prior to the release of the slave spacecraft, its attitude has to be initialized and the systematic errors of its gyroscope unit (GU) should be calibrated, such that the slave spacecraft can perform its space mission independently. The attitude determination accuracy of the slave spacecraft depends on the calibration accuracy of the systematic errors, such as gyroscope bias, scale factor error and misalignment [4,5,6]. Although the ground calibration method is convenient to achieve high accuracy [7,8], the values of the calibration parameters may change due to the difference between the ground and space environments [9]. The primary cause for the change in the misalignment, which is the most dominant error that affects the attitude determination accuracy, is the thermal distortion of the GU bracket and the spacecraft body, which yields a pointing change in the GU axis in the space environment. To cope with this problem, the attitude determination system (ADS) of the master spacecraft can be utilized as the reference for the on-orbit calibration [10]. Typically, the high-precision gyroscopes and the star sensors are mounted on the master spacecraft for attitude determination [11,12,13]. An accurate and rapid transfer alignment is crucial to guarantee the attitude determination performance of the slave spacecraft [14,15].

The main problem of the traditional inter-spacecraft transfer alignment method is that the convergence rate of the algorithm is rather slow. Typically, tens of minutes are required for the traditional transfer alignment process [16], which is not satisfactory for quick response situations. To achieve faster convergence and increased accuracy, the rapid transfer alignment techniques are investigated in this paper.

The measurement information matching scheme is an important research object in the field of transfer alignment. Many scholars focus on the on-orbit calibration method for the GU based on the attitude matching scheme [17,18]. The traditional method is to estimate the attitude and the calibration parameters of the GU on the slave spacecraft using the attitude measurement data provided by the star sensors on the master spacecraft with a Kalman filter (KF) [19,20]. The high-accuracy measurement information obtained from the satellite payload and GNSS (global navigation satellite system) are taken into account for the calibration [21,22]. In addition, for tactical weapon applications, the velocity plus attitude matching scheme [23,24] is developed as an improvement of the conventional velocity matching scheme [25,26,27]. The calibration methods based on position matching and acceleration matching schemes are studied in [28,29].

On the basis of measurement information matching, the performance of the inter-spacecraft transfer alignment method depends on the filtering algorithm. The KF is the most widely used state estimation approach for the space missions [30,31,32]. The estimation accuracy of the KF depends on the process and measurement noise covariance matrices ${\mathit{Q}}_k$ and ${\mathit{R}}_k$ in the system model. If the prior statistical characteristics of the process and measurement noises are inaccurate, the performance of the KF may be degraded evidently. For the inter-spacecraft transfer alignment system, the statistical characteristics of the gyroscopes and the star sensors can be achieved from the manufacturer, while the process noise covariance related to the calibration parameters are often not known exactly. To cope with the problem, the common approach is to estimate the unknown noise covariance matrix with an adaptive Kalman filter (AKF) [33,34]. In the AKF, the estimate of ${\mathit{Q}}_k$ is updated recursively based on the measurement innovation, which is calculated with the previous state estimate in each iteration. However, when the previous state estimate is inaccurate, and the prior statistical information is far from the actual situation, it is difficult to obtain the proper ${\mathit{Q}}_k$ with the AKF [35]. To the best of the authors’ knowledge, the optimal approach to tune the process noise covariance matrix is still an open question.

Motivated by the key idea of the transfer alignment method for tactical weapon applications and the AKF algorithm, to improve the accuracy and rapidity of the inter-spacecraft transfer alignment, a practical method is presented in this paper. The main contributions of the paper are given as follows:

(1)

An attitude plus angular rate matching scheme is presented, where the fused information from the star sensors and the gyroscopes on the master spacecraft is adopted for the calibration of the GU on the slave spacecraft. Accordingly, the state equation and the measurement equation are derived as the transfer alignment system model. Compared with the traditional attitude matching scheme, the main advantage of the presented scheme is that more measurement information is utilized, such that the alignment performance is improved in limited time.

(2)

A framework of the Q-learning Kalman filter (QKF) that combines the celebrated Q-learning approach [36,37,38] with the KF is developed to fine tune the process noise covariance matrix related to the calibration parameters. Instead of the recursive estimation in the AKF, the Q-learning approach is designed to explore for the appropriate ${\mathit{Q}}_k$. Once the appropriate ${\mathit{Q}}_k$ is learned, it is plugged into the model-based KF to enhance its performance. Compared with our previous works [39,40], the main advantage of the presented QKF is that only one explorative filter (instead of a group of parallel filters) is required in the Q-learning process so as to simplify the implementation of the algorithm. Correspondingly, the computational load of the algorithm is decreased, which facilitates the application of the algorithm on the spacecraft with limited computing resource.

This study is structured as follows. In Section 2, the attitude plus angular rate matching scheme and the inter-spacecraft rapid transfer alignment system model are presented. In Section 3, the QKF algorithm for rapid transfer alignment is provided. In Section 4, the potential performance of the transfer alignment system is analyzed via the calculation of the Cramer–Rao lower bounds (CRLB) [41]. In Section 5, the estimation performance of the KF, the AKF and the presented QKF designed based on the transfer alignment system model are compared via simulations. Finally, the conclusions are drawn in the last section.

2. Attitude Plus Angular Rate Matching Scheme

2.1. Main Idea

The inter-spacecraft rapid transfer alignment model is derived to calibrate the GU on the slave spacecraft based on the ADS on the master spacecraft. The basic principle of the considered transfer alignment system is illustrated in Figure 1.

As shown in Figure 1, on the master spacecraft, the ADS consisting of gyroscopes and star sensors are utilized to obtain the attitude reference information. On the slave spacecraft, the transfer alignment filter incorporates the attitude reference information and the GU measurement to estimate the attitude, the GU bias and the calibration parameters. The transfer alignment filter is designed based on the transfer alignment model composed of the state equation and the measurement equation. The calibration parameters include the scale factor error and the misalignment of the GU on the slave spacecraft. The construction of the transfer alignment model and the parameterization of the systematic errors fit within the spacecraft attitude determination framework [10].

In this paper, the spacecraft attitude describes the rotation of the spacecraft body frame relative to the geocentric equatorial inertial frame. For the geocentric equatorial inertial frame, the origin is the Earth’s center, the X axis points to the Equinox direction, the Y axis points toward the North pole and the Z axis forms a right-handed coordinate with the X axis and Y axis. For the spacecraft body frame, the origin is the center mass of the spacecraft, and the X axis, Y axis and Z axis are parallel to the principal axis of inertia and form a right-handed coordinate.

2.2. State Equation

For the design of the transfer alignment filter, the state equation of the transfer alignment model is established based on the attitude kinematics and the error model of the GU on the slave spacecraft. In this paper, the GU is supposed to be three gyroscopes which measure the angular rate of the slave spacecraft relative to the inertial frame. The GU error model includes gyroscope bias, scale factor error, misalignment and random noise composed of angle random walk (ARW) and rate random walk (RRW), as shown in Figure 2.

In the GU error model, the measured angular rate ${\mathit{\omega }}_{\mathrm{g}k}$ is related to the true angular rate ${\mathit{\omega }}_{\mathrm{b}k}$ of the slave spacecraft relative to inertial frame in the body frame by [21]

$${\mathit{\omega }}_{\mathrm{g}k}=\left(\mathbf{I}+{\mathit{\Lambda }}_k\right)\left(\mathbf{I}+{\mathit{\Delta }}_k\right){\mathbf{C}}_{\mathrm{b}}^{\mathrm{g}}{\mathit{\omega }}_{\mathrm{b}k}+{\mathit{b}}_k+{\mathit{\eta }}_{\mathrm{a}k},$$

(1)

where $\mathbf{I}$ denotes the unit matrix with compatible dimension, ${\mathit{\Lambda }}_k$ is the scale factor error matrix, ${\mathit{\Delta }}_k$ is the misalignment matrix, ${\mathbf{C}}_{\mathrm{b}}^{\mathrm{g}}$ is the attitude transformation matrix from the body frame to the sensor frame, ${\mathit{b}}_k$ is the GU bias, ${\mathit{\eta }}_{\mathrm{a}k}$ is the random noise called angular random walk, the subscript $k$ denotes discrete time. The true angular rate is written as ${\mathit{\omega }}_{\mathrm{b}k}={\left[\begin{array}{ccc}{\omega }_{\mathrm{b}\mathrm{x}k}& {\omega }_{\mathrm{b}\mathrm{y}k}& {\omega }_{\mathrm{b}\mathrm{z}k}\end{array}\right]}^{\mathrm{T}}$, where ${\omega }_{\mathrm{b}\mathrm{x}k}$, ${\omega }_{\mathrm{b}\mathrm{y}k}$ and ${\omega }_{\mathrm{b}\mathrm{z}k}$ are the elements of 3-axis angular rate. The scale factor error matrix is given by

$${\mathit{\Lambda }}_k=\left[\begin{array}{ccc}{\lambda }_{\mathrm{x}k}& 0& 0\\ 0& {\lambda }_{\mathrm{y}k}& 0\\ 0& 0& {\lambda }_{\mathrm{z}k}\end{array}\right],$$

(2)

where ${\lambda }_{\mathrm{x}k}$, ${\lambda }_{\mathrm{y}k}$ and ${\lambda }_{\mathrm{z}k}$ are the scale factor error parameters. The misalignment matrix is described as

$${\mathit{\Delta }}_k=\left[\begin{array}{ccc}0& {\overline{\delta }}_{\mathrm{x}\mathrm{y}k}& {\overline{\delta }}_{\mathrm{x}\mathrm{z}k}\\ {\overline{\delta }}_{\mathrm{y}\mathrm{x}k}& 0& {\overline{\delta }}_{\mathrm{y}\mathrm{z}k}\\ {\overline{\delta }}_{\mathrm{z}\mathrm{x}k}& {\overline{\delta }}_{\mathrm{z}\mathrm{y}k}& 0\end{array}\right],$$

(3)

where ${\overline{\delta }}_{\mathrm{x}\mathrm{y}k}$, ${\overline{\delta }}_{\mathrm{x}\mathrm{z}k}$, ${\overline{\delta }}_{\mathrm{y}\mathrm{x}k}$, ${\overline{\delta }}_{\mathrm{y}\mathrm{z}k}$, ${\overline{\delta }}_{\mathrm{z}\mathrm{x}k}$ and ${\overline{\delta }}_{\mathrm{z}\mathrm{y}k}$ are elements in the misalignment matrix. From Equations (2) and (3), we have

$${(\mathbf{I}+\mathit{\Lambda }}_k)\left(\mathbf{I}+{\mathit{\Delta }}_k\right)={\mathbf{I}+\mathit{M}}_k,$$

(4)

with

$${\mathit{M}}_k=\left[\begin{array}{ccc}{\lambda }_{\mathrm{x}k}& {\delta }_{\mathrm{x}\mathrm{y}k}& {\delta }_{\mathrm{x}\mathrm{z}k}\\ {\delta }_{\mathrm{y}\mathrm{x}k}& {\lambda }_{\mathrm{y}k}& {\delta }_{\mathrm{y}\mathrm{z}k}\\ {\delta }_{\mathrm{z}\mathrm{x}k}& {\delta }_{\mathrm{z}\mathrm{y}k}& {\lambda }_{\mathrm{z}k}\end{array}\right],$$

(5)

where

$${\delta }_{\mathrm{x}\mathrm{y}k}=(1+{\lambda }_{\mathrm{x}k}){\overline{\delta }}_{\mathrm{x}\mathrm{y}k},$$

(6)

$${\delta }_{\mathrm{x}\mathrm{z}k}=(1+{\lambda }_{\mathrm{x}k}){\overline{\delta }}_{\mathrm{x}\mathrm{z}k},$$

(7)

$${\delta }_{\mathrm{y}\mathrm{x}k}=\left(1+{\lambda }_{\mathrm{y}k}\right){\overline{\delta }}_{\mathrm{y}\mathrm{x}k},$$

(8)

$${\delta }_{\mathrm{y}\mathrm{z}k}=(1+{\lambda }_{\mathrm{y}k}){\overline{\delta }}_{\mathrm{y}\mathrm{z}k},$$

(9)

$${\delta }_{\mathrm{z}\mathrm{x}k}=(1+{\lambda }_{\mathrm{z}k}){\overline{\delta }}_{\mathrm{z}\mathrm{x}k},$$

(10)

$${\delta }_{\mathrm{z}\mathrm{y}k}=\left(1+{\lambda }_{\mathrm{z}k}\right){\overline{\delta }}_{\mathrm{z}\mathrm{y}k}.$$

(11)

${\delta }_{\mathrm{x}\mathrm{y}k}$, ${\delta }_{\mathrm{x}\mathrm{z}k}$, ${\delta }_{\mathrm{y}\mathrm{x}k}$, ${\delta }_{\mathrm{y}\mathrm{z}k}$, ${\delta }_{\mathrm{z}\mathrm{x}k}$ and ${\delta }_{\mathrm{z}\mathrm{y}k}$ are the misalignment parameters. Note that the products of scale factors and misalignments are combined in Equation (5) to simplify the formulation. Substituting Equation (4) into Equation (1), the GU error model is reformulated as

$${\mathit{\omega }}_{\mathrm{g}k}=\left({\mathbf{I}+\mathit{M}}_k\right){\mathbf{C}}_{\mathrm{b}}^{\mathrm{g}}{\mathit{\omega }}_{\mathrm{b}k}+{\mathit{b}}_k+{\mathit{\eta }}_{\mathrm{a}k}.$$

(12)

To derive the attitude error equation, the estimate of the spacecraft angular rate ${\widehat{\mathit{\omega }}}_{bk}$ is written as

$${\widehat{\mathit{\omega }}}_{\mathrm{b}k}={\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}({\mathit{\omega }}_{\mathrm{g}k}-{\widehat{\mathit{b}}}_k),$$

(13)

where ${\widehat{\mathit{b}}}_k$ is the estimate of the GU bias, ${\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}$ is the attitude transformation matrix from the sensor frame to the body frame. Substituting Equation (12) into Equation (13), we have

$${\widehat{\mathit{\omega }}}_{\mathrm{b}k}={\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}\left({\mathbf{I}+\mathit{M}}_k\right){\mathbf{C}}_{\mathrm{b}}^{\mathrm{g}}{\mathit{\omega }}_{\mathrm{b}k}+{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\delta \mathit{b}}_k+{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\mathit{\eta }}_{\mathrm{a}k},$$

(14)

where

$${\delta \mathit{b}}_k={\mathit{b}}_k-{\widehat{\mathit{b}}}_k,$$

(15)

is the GU bias error. The angular rate estimation error is defined as

$${\delta \mathit{\omega }}_{\mathrm{b}k}={\mathit{\omega }}_{\mathrm{b}k}-{\widehat{\mathit{\omega }}}_{\mathrm{b}k}.$$

(16)

Substituting Equation (14) into Equation (16) yields

$${\delta \mathit{\omega }}_{\mathrm{b}k}=-{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\mathit{M}}_k{\mathbf{C}}_{\mathrm{b}}^{\mathrm{g}}{\mathit{\omega }}_{\mathrm{b}k}-{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\delta \mathit{b}}_k-{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\mathit{\eta }}_{\mathrm{a}k}.$$

(17)

Equation (17) is reformulated as

$${\delta \mathit{\omega }}_{\mathrm{b}k}=-{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\mathit{\Omega }}_{\mathrm{g}k}{\mathit{\delta }}_k-{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\delta \mathit{b}}_k-{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\mathit{\eta }}_{\mathrm{a}k},$$

(18)

with

$${\mathit{\Omega }}_{\mathrm{g}k}=\left[\begin{array}{ccc}{\left({\mathbf{C}}_{\mathrm{b}}^{\mathrm{g}}{\mathit{\omega }}_{\mathrm{b}k}\right)}^{\mathrm{T}}& {\mathbf{0}}_{1\times 3}& {\mathbf{0}}_{1\times 3}\\ {\mathbf{0}}_{1\times 3}& {\left({\mathbf{C}}_{\mathrm{b}}^{\mathrm{g}}{\mathit{\omega }}_{\mathrm{b}k}\right)}^{\mathrm{T}}& {\mathbf{0}}_{1\times 3}\\ {\mathbf{0}}_{1\times 3}& {\mathbf{0}}_{1\times 3}& {\left({\mathbf{C}}_{\mathrm{b}}^{\mathrm{g}}{\mathit{\omega }}_{\mathrm{b}k}\right)}^{\mathrm{T}}\end{array}\right] ,$$

(19)

and

$${\mathit{\delta }}_k={\left[\begin{array}{ccc}\begin{array}{ccc}{\lambda }_{\mathrm{x}k}& {\delta }_{\mathrm{x}\mathrm{y}k}& {\delta }_{\mathrm{x}\mathrm{z}k}\end{array}& \begin{array}{ccc}{\delta }_{\mathrm{y}\mathrm{x}k}& {\lambda }_{\mathrm{y}k}& {\delta }_{\mathrm{y}\mathrm{z}k}\end{array}& \begin{array}{ccc}{\delta }_{\mathrm{z}\mathrm{x}k}& {\delta }_{\mathrm{z}\mathrm{y}k}& {\lambda }_{\mathrm{z}k}\end{array}\end{array}\right]}^{\mathrm{T}}$$

(20)

is the calibration parameter vector.

For the spacecraft attitude determination, the error quaternion $\mathsf{\delta }{\mathit{q}}_k$ is defined as

$$\mathsf{\delta }{\mathit{q}}_k={\mathit{q}}_k⨂{\widehat{\mathit{q}}}_k^{-1},$$

(21)

where ${\mathit{q}}_k$ is the attitude quaternion that describes the spacecraft attitude relative to the inertial frame, ${\widehat{\mathit{q}}}_k$ is the estimate of ${\mathit{q}}_k$. According to the spacecraft attitude kinematics, the propagation of the error quaternion is described by the following perturbation equation [11]:

$$\mathsf{\delta }{\mathit{\rho }}_{k+1}=\left(\mathbf{I}-\mathsf{\tau }\left[{\mathit{\omega }}_{\mathrm{b}k}\times \right]\right)\mathsf{\delta }{\mathit{\rho }}_k+{\displaystyle \frac{\mathsf{\tau }}{2}}{\delta \mathit{\omega }}_{\mathrm{b}k},$$

(22)

where $\mathsf{\delta }{\mathit{\rho }}_k$ is the vector part of the error quaternion, $\mathsf{\tau }$ denotes the time interval of discretization, $\left[{\mathit{\omega }}_{\mathrm{b}k}\times \right]$ is the skew symmetric matrix defined as

$$\left[{\mathit{\omega }}_{\mathrm{b}k}\times \right]=\left[\begin{array}{ccc}0& -{\omega }_{\mathrm{b}\mathrm{z}k}& {\omega }_{\mathrm{b}\mathrm{y}k}\\ {\omega }_{\mathrm{b}\mathrm{z}k}& 0& -{\omega }_{\mathrm{b}\mathrm{x}k}\\ -{\omega }_{\mathrm{b}\mathrm{y}k}& {\omega }_{\mathrm{b}\mathrm{x}k}& 0\end{array}\right].$$

(23)

Substituting Equation (18) into Equation (22), the perturbation equation is modified as

$$\mathsf{\delta }{\mathit{\rho }}_{k+1}=\left(\mathbf{I}-\mathsf{\tau }\left[{\mathit{\omega }}_{\mathrm{b}k}\times \right]\right)\mathsf{\delta }{\mathit{\rho }}_k-{\displaystyle \frac{\mathsf{\tau }}{2}}{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\mathit{\Omega }}_{\mathrm{g}k}{\mathit{\delta }}_k-{\displaystyle \frac{\mathsf{\tau }}{2}}{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\delta \mathit{b}}_k-{\displaystyle \frac{\mathsf{\tau }}{2}}{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\mathit{\eta }}_{\mathrm{a}k}.$$

(24)

It is evident that the effects of the GU bias, scale factor error, misalignment and random noise to the propagation of the vector part of the error quaternion are described in Equation (24).

For transfer alignment, the spacecraft attitude, the GU bias, scale factor error and misalignment should be estimated. Accordingly, the state vector is constructed as the combination of the vector part of the error quaternion $\mathsf{\delta }{\mathit{\rho }}_k$, the GU bias error ${\delta \mathit{b}}_k$ and the calibration parameter vector ${\mathit{\delta }}_k$, which is given by

$${\mathit{x}}_k={\left[\begin{array}{ccc}{\delta \mathit{\rho }}_k^{\mathrm{T}}& \delta {\mathit{b}}_k^{\mathrm{T}}& {\mathit{\delta }}_k^{\mathrm{T}}\end{array}\right]}^{\mathrm{T}}.$$

(25)

From Equations (24) and (25), we obtain the state equation:

$${\mathit{x}}_{k+1}={\mathit{F}}_k{\mathit{x}}_k+{\mathit{w}}_k,$$

(26)

with the state transition matrix

$${\mathit{F}}_k=\left[\begin{array}{ccc}{\mathbf{I}}_{3\times 3}-\mathsf{\tau }\left[{\mathit{\omega }}_{\mathrm{b}k}\times \right]& -{\displaystyle \frac{\mathsf{\tau }}{2}}{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}& -{\displaystyle \frac{\mathsf{\tau }}{2}}{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\mathit{\Omega }}_{\mathrm{g}k}\\ {\mathbf{0}}_{3\times 3}& {\mathbf{I}}_{3\times 3}& {\mathbf{0}}_{3\times 9}\\ {\mathbf{0}}_{9\times 3}& {\mathbf{0}}_{9\times 3}& {\mathbf{I}}_{9\times 9}\end{array}\right].$$

(27)

The process noise ${\mathit{w}}_k$ is given by

$${\mathit{w}}_k={\left[\begin{array}{ccc}-{\displaystyle \frac{\mathsf{\tau }}{2}}{\left({\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\mathit{\eta }}_{\mathrm{a}k}\right)}^{\mathrm{T}}& {\mathit{\eta }}_{\mathrm{r}k}^{\mathrm{T}}& {\mathit{\eta }}_{\mathrm{c}k}^{\mathrm{T}}\end{array}\right]}^{\mathrm{T}},$$

(28)

where ${\mathit{\eta }}_{\mathrm{r}k}$ is the random noise that drive the rate random walk, ${\mathit{\eta }}_{\mathrm{c}k}$ is introduced to describe the drift of the calibration parameters. It is often assumed that ${\mathit{w}}_k$ is the Gaussian white noise with zero mean. The process noise covariance matrix ${\mathit{Q}}_k$ is a positive definite symmetric matrix with the following structure

$${\mathit{Q}}_k=\mathrm{E}\left(\left[\begin{array}{c}-{\displaystyle \frac{\mathsf{\tau }}{2}}{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\mathit{\eta }}_{\mathrm{a}k}\\ {\mathit{\eta }}_{\mathrm{r}k}\\ {\mathit{\eta }}_{\mathrm{c}k}\end{array}\right]\left[\begin{array}{ccc}-{\displaystyle \frac{\mathsf{\tau }}{2}}{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\mathit{\eta }}_{\mathrm{a}k}& {\mathit{\eta }}_{\mathrm{r}k}& {\mathit{\eta }}_{\mathrm{c}k}\end{array}\right]\right)=\left[\begin{array}{ccc}{\mathit{Q}}_{ak}& 0& 0\\ 0& {\mathit{Q}}_{rk}& 0\\ 0& 0& {\mathit{Q}}_{ck}\end{array}\right],$$

(29)

where ${\mathit{Q}}_{ak}$ and ${\mathit{Q}}_{rk}$ are the sub-matrices related to the vector part of the error quaternion $\mathsf{\delta }{\mathit{\rho }}_k$ and the GU bias error ${\delta \mathit{b}}_k$ respectively, ${\mathit{Q}}_{ck}$ is the sub-matrix related to the calibration parameter vector ${\mathit{\delta }}_k$.

2.3. Measurement Model

For the attitude plus angular rate matching scheme, both the attitude and the angular rate reference information achieved from the ADS on the master spacecraft are utilized for the on-orbit calibration. It is expected that the calibration parameters can be estimated effectively with the attitude reference information from the master spacecraft. When the attitude and the angular reference information is available, the measurement equation is written as

$${\mathit{y}}_k={\mathit{H}}_k{\mathit{x}}_k+{\mathit{v}}_k,$$

(30)

with the measurement

$${\mathit{y}}_k={\left[\begin{array}{cc}{\mathit{y}}_{\rho k}^{\mathrm{T}}& {\mathit{y}}_{\omega k}^{\mathrm{T}}\end{array}\right]}^{\mathrm{T}},$$

(31)

the measurement matrix

$${\mathit{H}}_k=\left[\begin{array}{ccc}{\mathbf{I}}_{3\times 3}& {\mathbf{0}}_{3\times 3}& {\mathbf{0}}_{3\times 9}\\ {\mathbf{0}}_{3\times 3}& -{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}& -{\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\mathit{\Omega }}_{\mathrm{g}k}\end{array}\right],$$

(32)

and the measurement noise

$${\mathit{v}}_k={\left[\begin{array}{cc}{\mathit{v}}_{\rho k}^{\mathrm{T}}& {\mathit{v}}_{\omega k}^{\mathrm{T}}\end{array}\right]}^{\mathrm{T}},$$

(33)

where ${\mathit{y}}_{\rho k}$ is the vector part of the error quaternion between the quaternions obtained from the ADS on the master spacecraft and the GU on the slave spacecraft, ${\mathit{y}}_{\omega k}$ is the difference between the angular rates obtained from the ADS on the master spacecraft and the GU on the slave spacecraft, ${\mathit{v}}_{\rho k}$ is the attitude measurement noise with the covariance matrix ${\mathit{R}}_{\rho k}$, ${\mathit{v}}_{\omega k}$ is the angular rate measurement noise with the covariance matrix ${\mathit{R}}_{\omega k}$. The measurement noise covariance matrix ${\mathit{R}}_k$ is the mixture of ${\mathit{R}}_{\rho k}$ and ${\mathit{R}}_{\omega k}$:

$${\mathit{R}}_k=\left[\begin{array}{cc}{\mathit{R}}_{\rho k}& \mathbf{0}\\ \mathbf{0}& {\mathit{R}}_{\omega k}\end{array}\right].$$

(34)

Note that both the measurement noises of the ADS on the master spacecraft and the GU on the slave spacecraft are contained in Equation (33). Generally, to implement the inter-spacecraft rapid transfer alignment effectively, the ADS on the master spacecraft should be more accurate than the GU on the slave spacecraft. It is expected that the systematic errors in the ADS on the master spacecraft have been compensated before the implementation of the inter-spacecraft rapid transfer alignment.

The state Equation (26) and the measurement Equation (30) compose the system model for the attitude plus angular rate matching scheme. With the system model, the transfer alignment KF is designed to implement the state estimation. It should be mentioned that, for the considered attitude determination system composed of the star sensors and the gyroscopes, as it is widely used in current satellites, the feasibility of the system model has been verified through multiple space missions. In general, for a novel navigation system, the hardware-in-loop experiment with the simulation of the operational environment is an effective approach for model verification.

2.4. Transfer Alignment KF

On the basis of the system model, the transfer alignment KF is designed to estimate the state vector ${\mathit{x}}_k$ based on the measurement ${\mathit{y}}_k$. The procedure of the inter-spacecraft rapid transfer alignment method based on the KF with the prediction and update procedures is collected in Algorithm 1.

Algorithm 1: Transfer alignment Kalman filter

1: Initialize attitude quaternion estimate ${\widehat{\mathit{q}}}_0$, bias estimate ${\widehat{\mathit{b}}}_0$, state estimate ${\widehat{\mathit{x}}}_0$ and its estimation error covariance matrix ${\mathit{P}}_0$

2: for k = 1, 2, …, K, do

3: ${\overline{\mathsf{\omega }}}_{\mathrm{b}k-1}\leftarrow {\mathbf{C}}_{\mathrm{g}}^{\mathrm{b}}{\left({\mathit{I}+\widehat{\mathit{M}}}_{k-1}\right)}^{-1}({\mathit{\omega }}_{\mathrm{g}k-1}-{\widehat{\mathit{b}}}_{k-1})$

4: ${\widehat{\mathit{q}}}_k\leftarrow [{\mathbf{I}}_{4\times 4}\cos \left({\displaystyle \frac{‖{\overline{\mathit{\omega }}}_{\mathrm{b}k-1}‖\mathsf{\tau }}{2}}\right)+{\displaystyle \frac{\mathit{\Phi }\left({\overline{\mathit{\omega }}}_{\mathrm{b}k-1}\right)}{‖{\overline{\mathsf{\omega }}}_{\mathrm{b}k-1}‖}}\mathrm{s}\mathrm{i}\mathrm{n}({\displaystyle \frac{‖{\overline{\mathit{\omega }}}_{\mathrm{b}k-1}‖\mathsf{\tau }}{2}}\left)\right]{\widehat{\mathit{q}}}_{k-1}$

5: ${\widehat{\mathit{b}}}_k\leftarrow {\widehat{\mathit{b}}}_{k-1}$

6: ${\widehat{\mathit{x}}}_{k|k-1}(1:\mathrm{6,1})\leftarrow \mathbf{0}$

7: ${\widehat{\mathit{x}}}_{k|k-1}(7:\mathrm{15,1})\leftarrow {\widehat{\mathit{x}}}_{k-1}(7:\mathrm{15,1})$

8: ${\mathit{P}}_{k|k-1}\leftarrow {\mathit{F}}_k{\mathit{P}}_{k-1}{\mathit{F}}_k^{\mathrm{T}}+{\mathit{Q}}_k$

9: if the measurement ${\mathit{y}}_k$ is available, then

10:  ${\mathit{K}}_k\leftarrow {\mathit{P}}_{k|k-1}{\mathit{H}}_k^{\mathrm{T}}({\mathit{H}}_k{\mathit{P}}_{k|k-1}{\mathit{H}}_k^{\mathrm{T}}{+{\mathit{R}}_k)}^{-1}$

11:  ${\tilde{\mathit{y}}}_k\leftarrow {\mathit{y}}_k-{\mathit{H}}_k{\widehat{\mathit{x}}}_{k|k-1}$

12:  ${\widehat{\mathit{x}}}_k\leftarrow {\widehat{\mathit{x}}}_{k|k-1}+{\mathit{K}}_k{\tilde{\mathit{y}}}_k$

13:  ${\mathit{P}}_k\leftarrow \left(\mathbf{I}-{\mathit{K}}_k{\mathit{H}}_k\right){\mathit{P}}_{k|k-1}{\left(\mathbf{I}-{\mathit{K}}_k{\mathit{H}}_k\right)}^{\mathrm{T}}+{\mathit{K}}_k{\mathit{R}}_k{\mathit{K}}_k^{\mathrm{T}}$

14: end if

15: ${\widehat{\mathit{q}}}_k\leftarrow {\delta \widehat{\mathit{q}}}_k⨂{\widehat{\mathit{q}}}_k$

16: ${\widehat{\mathit{b}}}_k\leftarrow {\widehat{\mathit{b}}}_k+{\delta \widehat{\mathit{b}}}_k$

17: end for

18: return $\left\{{\widehat{\mathit{q}}}_k\right\}$, $\left\{{\widehat{\mathit{b}}}_k\right\}$, $\left\{{\widehat{\mathit{x}}}_k\right\}$, $\left\{{\mathit{P}}_k\right\}$

In the algorithm, ${\widehat{\mathit{x}}}_{k|k-1}$ and ${\widehat{\mathit{x}}}_k$ are the prediction and the estimate of the state vector, ${\mathit{P}}_{k|k-1}$ and ${\mathit{P}}_k$ are their corresponding estimation error covariance matrices, K is the length of the measurement data, ${\mathit{K}}_k$ is the Kalman gain, ${\tilde{\mathit{y}}}_k$ is the measurement innovation. The matrix ${\widehat{\mathit{M}}}_k$ is constructed with the estimate of the calibration parameter vector as the estimate of ${\mathit{M}}_k$. The expression of $\mathit{\Phi }\left({\overline{\mathit{\omega }}}_{\mathrm{b}k}\right)$ is

$$\mathit{\Phi }\left({\overline{\mathit{\omega }}}_{\mathrm{b}k}\right)=\left[\begin{array}{cc}-[{\overline{\mathit{\omega }}}_{\mathrm{b}k}\times ]& {\overline{\mathit{\omega }}}_{\mathrm{b}k}\\ -{\overline{\mathit{\omega }}}_{\mathrm{b}k}^{\mathrm{T}}& 0\end{array}\right].$$

(35)

To facilitate the implementation of the algorithm, similar to the method presented in [11], as the attitude quaternion ${\widehat{\mathit{q}}}_k$ is propagated based on the spacecraft attitude kinematics equation, the state transition matrix ${\mathit{F}}_k$ in the state Equation (26) is only used for the estimation error covariance propagation in the KF. Note that the propagation of the error quaternion ${\delta \widehat{\mathit{q}}}_k$ is not beneficial to improve the filtering performance. To deal with the problem caused by the discontinuity of the measurement data, for the filtering algorithm shown in Algorithm 1, the propagation is performed in each time step, while the update is performed in the case that the measurement data are available.

It is seen from Algorithm 1 that the efficiency of the update to the prediction ${\widehat{\mathit{x}}}_{k|k-1}$ with the measurement innovation ${\tilde{\mathit{y}}}_k$ depends on the Kalman gain ${\mathit{K}}_k$, which is adjusted through the noise covariance matrices ${\mathit{Q}}_k$ and ${\mathit{R}}_k$. In the measurement noise covariance matrix, ${\mathit{R}}_{\rho k}$ and ${\mathit{R}}_{\omega k}$ are symmetric positive definite matrices determined according to the measurement error behavior of the star sensors and the GU specified by the manufacturer. In the process noise covariance matrix, the elements related to ${\mathit{\eta }}_{\mathrm{a}k}$ and ${\mathit{\eta }}_{\mathrm{r}k}$ are determined with the ARW and RRW coefficients, which are achievable through the Allan variance analysis [42]. However, it is difficult to determine the accurate noise covariance for the calibration parameters ${\mathit{\eta }}_{\mathrm{c}k}$ in the absence of prior knowledge. As mentioned in the introduction, the proper choice for the process noise covariance matrix is critical for accurate transfer alignment. The state estimate of the KF may deviate from its actual value if ${\mathit{Q}}_k$ is not set appropriately. In order to identify the unknown elements in ${\mathit{Q}}_k$, a Q-learning-based filtering algorithm is presented in the next section as a modification of the standard KF.

3. Q-Learning Kalman Filter

3.1. Q-Learning Approach

When aiming to solve the problem of fine tuning the elements related to the calibration parameters in the process noise covariance matrix of the transfer alignment KF, a Q-learning Kalman filter is presented, which is a combination of the KF algorithm and the Q-learning approach. The Q-learning approach is a representative reinforce learning (RL) method [43], which is constructed on the key idea that the successful decision should be remembered by the agent interacting with its environment, which could provide a reinforcement signal as the feedback to the successful decision of the agent, such that the decision becomes more likely to be made in the future.

The Q-learning approach is implemented through a recursive trial-and-error process. In each iteration, the agent performs an action and receives an immediate reward from the environment, which indicates whether the action is good or not. The reward is accumulated to update the action selection strategy represented with the Q-function. Then, an optimized action is selected according to the updated strategy for the next iteration. This process is repeated several times so that the appropriate action for the maximization of the accumulated reward tends to be selected by the agent. The model of the environment is not necessary in the Q-learning process, which facilitates the implementation of the approach. The Q-learning approach has become the basis of many learning algorithms and exhibits an excellent learning ability.

The purpose of the Q-learning approach is to achieve the proper strategy to select the action $a\in \mathbf{A}$ in a specific state $s\in \mathbf{S}$, where $\mathbf{A}$ and $\mathbf{S}$ are the action space and the state space, respectively. To achieve the purpose, the iterative update of the Q-function $Q^{\left(k\right)}(s,a)$ is formulated as

$$Q^{\left(k\right)}\left(s,a\right)=\left(1-\alpha \right)Q^{\left(k-1\right)}\left(s,a\right)+\mathsf{\alpha }\left[R\left(s,a\right)+\mathsf{\gamma }\underset{a^{’}\in \mathbf{A}}{\max }{Q}^{\left(k-1\right)}\left(s^{\prime },a^{\prime }\right)\right],$$

(36)

where $R\left(s,a\right)$ is the immediate reward for the action $a$ that is performed in the state $s$, $0<\mathsf{\alpha }\le 1$ is the learning rate, $0<\mathsf{\gamma }\le 1$ is the discount factor and $s’$ is the transited state after the action $a$ is performed. It is seen from (36) that the Q-learning process is the incremental estimation of the Q-function $Q^{\left(k\right)}(s,a)$. In the learning process, the discount factor $\mathsf{\gamma }$ is used to weigh between the immediate reward $R\left(s,a\right)$ and the accumulated reward represented by the previous estimation of the Q-function. The effect of the accumulated reward to the estimation of Q-function will be enhanced with the increase in the parameter $\mathsf{\gamma }$. The learning rate $\mathsf{\alpha }$ is used to weigh between the previous estimation of the Q-function $Q^{\left(k-1\right)}(s,a)$ and its update, as shown in (36). The effect of the update will be enhanced with the increase of the parameter $\mathsf{\alpha }$.

To guarantee the efficiency of the Q-learning approach, it is important to design the action space $\mathbf{A}$, the state space $\mathbf{S}$ and the reward $R\left(s,a\right)$ properly. According to the Q-function $Q^{\left(k\right)}(s,a)$, the selected action $a_{max}$ for the next iteration can be determined as

$$a_{max}\leftarrow \arg \underset{a\in \mathbf{A}}{\max }{Q}^{\left(k\right)}(s,a).$$

(37)

As the Q-function represents the accumulated reward, the action selection strategy shown in Equation (37) is beneficial to maximize the accumulated reward.

3.2. Q-Learning Based Covariance Tuning

The model-based KF and the data-driven Q-learning approach are combined to design the QKF algorithm, where the Q-learning approach is used instead of the recursive estimation in the AKF algorithm to determine the process noise covariance matrix. To simplify the Q-learning process, only the sub-matrix ${\mathit{Q}}_{ck}$ is tuned in the presented algorithm. The suggested framework of the QKF is shown in Figure 3.

The Q-learning approach is implemented based on the measurement data obtained from the ADS on the master spacecraft and the GU on the slave spacecraft. Once the sub-matrix corresponding to the calibration parameter vector is selected as ${\widehat{\mathit{Q}}}_{ck}$ via the Q-learning approach, it is plugged into the transfer alignment KF presented in Algorithm 1 for the state estimation. The process noise covariance matrix of the transfer alignment KF is formulated as

$${\widehat{\mathit{Q}}}_k=\left[\begin{array}{ccc}{\mathit{Q}}_{ak}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathit{Q}}_{rk}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\widehat{\mathit{Q}}}_{ck}\end{array}\right].$$

(38)

It differs from that of the standard KF as the fine-tuned ${\widehat{\mathit{Q}}}_k$ is adopted instead of the original ${\mathit{Q}}_k$ to calculate the gain matrix ${\mathit{K}}_k$. In other application scenarios, all sub-matrices in ${\mathit{Q}}_k$ should be tuned, a parallel Q-learning based filtering algorithm similar to the method developed in [40] could be adopted.

To select the appropriate process noise covariance matrix for filtering performance enhancement, in the designed Q-learning approach, the state space $\mathbf{S}$ is constructed with different design values of the sub-matrix ${\mathit{Q}}_{ck}^{\left(s\right)}$. Each state $s$ is related to an element in the pre-determined set $\left\{\cdots ,{\mathit{Q}}_{ck}^{\left(s\right)},\cdots \right\}$. The action space $\mathbf{A}$ is constructed with different state transition actions, including the transition to the adjacent state or stay at the current state. Furthermore, the immediate reward $R\left(s,a\right)$ is constructed with the measurement innovation of an explorative KF designed based on the system model shown in (26) and (30). The structure and parameters of the explorative KF is similar to those of the transfer alignment KF, except the process noise covariance matrix related to the current state ${\overline{\mathit{Q}}}_k^{\left(s\right)}$ is adopted. Specifically, the process noise covariance matrix in the explorative KF is formulated as

$${\overline{\mathit{Q}}}_k^{\left(s\right)}=\left[\begin{array}{ccc}{\mathit{Q}}_{ak}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathit{Q}}_{rk}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathit{Q}}_{ck}^{\left(s\right)}\end{array}\right].$$

(39)

In the Q-learning process, for the state that transits from $s$ to $s’$ after the action $a$ is performed, the reward is designed as

$$R\left(s,a\right)={\left({\tilde{\mathit{y}}}_e^{\left(s\right)}\right)}^{\mathrm{T}}{\tilde{\mathit{y}}}_e^{\left(s\right)}-{\left({\tilde{\mathit{y}}}_e^{(s’)}\right)}^{\mathrm{T}}{\tilde{\mathit{y}}}_e^{(s’)},$$

(40)

where ${\tilde{\mathit{y}}}_e^{\left(s\right)}$ is the measurement innovation of the explorative KF with ${\overline{\mathit{Q}}}_k^{\left(s\right)}$ used instead of ${\mathit{Q}}_k$. From the reward represented in Equation (40), it is evident that a positive reward is achieved if the action $a$ is beneficial to reduce the measurement innovation. Conversely, the action that increases the measurement innovation leads to negative feedback for the agent. Considering that measurement innovation is an indicator of filtering performance, it is expected that the sub-matrix ${\widehat{\mathit{Q}}}_{ck},$ selected with the Q-learning approach via the maximization of the cumulative reward represented by the Q-function $Q\left(s,a\right),$ is valuable to improve the state estimation accuracy. Note that the subscript $\left(k\right)$ of the Q-function $Q^{\left(k\right)}(s,a)$ is omitted hereafter to simplify the notation.

Following the previous description, the Q-learning process in the QKF algorithm to achieve ${\widehat{\mathit{Q}}}_{ck}$ is presented in Algorithm 2. To balance the exploration and exploitation in the Q-learning process, the $\mathsf{\epsilon }$–greedy strategy [43] is adopted as a modification of the basic action selection strategy shown in Equation (37).

Algorithm 2: Q-learning process in QKF

1: ${\widehat{\mathit{x}}}_{\mathrm{e}0}\leftarrow {\widehat{\mathit{x}}}_0$, ${\mathit{P}}_{\mathrm{e}0}\leftarrow {\mathit{P}}_0$

2: Initialize parameter set $\left\{\cdots ,{\mathit{Q}}_{ck}^{\left(s\right)},\cdots \right\}$

3: for all $s\in \mathbf{S}$, ${\overline{\mathit{Q}}}_k^{\left(s\right)}(1:\mathrm{6,1}:6)\leftarrow {\mathit{Q}}_k(1:\mathrm{6,1}:6)$, ${\overline{\mathit{Q}}}_k^{\left(s\right)}(7:\mathrm{15,7}:15)\leftarrow {\mathit{Q}}_{ck}^{\left(s\right)}$

4: for all $s\in \mathbf{S}$, $a\in \mathbf{A}$, $Q(s,a)\leftarrow 0$

5: Initialize state $s$

6: ${\tilde{\mathit{y}}}_e^{\left(s\right)}\leftarrow \mathrm{s}\mathrm{q}\mathrm{r}\mathrm{t}\left(\mathrm{e}\mathrm{i}\mathrm{g}\left({\mathit{H}}_0{\mathit{P}}_{\mathrm{e}0}{\mathit{H}}_0^{\mathrm{T}}+{\mathit{R}}_0\right)\right)$

7: for k = 1, 2, …, K, do

8:  $a\leftarrow \mathsf{\epsilon }–\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{e}\mathrm{d}\mathrm{y}(s,Q(s,a),\mathbf{A},\mathsf{\epsilon })$

9:  Perform $a$ and observe reached state $s’$

10:  $[{\widehat{\mathit{x}}}_{\mathrm{e}k},{\mathit{P}}_{\mathrm{e}k},{\tilde{\mathit{y}}}_e^{(s’)}]\leftarrow \mathrm{K}\mathrm{F}({\widehat{\mathit{x}}}_{\mathrm{e}k-1},{\mathit{P}}_{\mathrm{e}k-1},{\mathit{y}}_k,{\overline{\mathit{Q}}}_k^{(s’)},{\mathit{R}}_k)$

11:  $R\left(s,a\right)\leftarrow {\left({\tilde{\mathit{y}}}_e^{\left(s\right)}\right)}^{\mathrm{T}}{\tilde{\mathit{y}}}_e^{\left(s\right)}-{\left({\tilde{\mathit{y}}}_e^{(s’)}\right)}^{\mathrm{T}}{\tilde{\mathit{y}}}_e^{(s’)}$

12:  $Q\left(s,a\right)=\left(1-\alpha \right)Q\left(s,a\right)+\mathsf{\alpha }[R\left(s,a\right)+\mathsf{\gamma }\underset{\mathit{a’}\in \mathbf{A}}{\max }Q\left(s’,a’\right)]$

13:  $s\leftarrow s’$

14:  ${\tilde{\mathit{y}}}_e^{\left(s\right)}\leftarrow {\tilde{\mathit{y}}}_e^{(s’)}$

15: end for

16: ${\widehat{\mathit{Q}}}_{ck}\leftarrow {\mathit{Q}}_{ck}^{\left(s\right)}$

17: return ${\widehat{\mathit{Q}}}_{ck}$

In the algorithm, ${\widehat{\mathit{x}}}_{\mathrm{e}k}$ and ${\mathit{P}}_{\mathrm{e}k}$ are the state estimate and its estimation error covariance matrix of the explorative KF, and $\mathsf{\epsilon }$ is the probability of the $\mathsf{\epsilon }$-greedy strategy to select a random action. The function $\mathrm{e}\mathrm{i}\mathrm{g}\left(\right)$ denotes the eigenvalues of a square matrix. The function $\mathrm{s}\mathrm{q}\mathrm{r}\mathrm{t}\left(\right)$ denotes the square root of each element in a vector. The function $\mathsf{\epsilon }–\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{e}\mathrm{d}\mathrm{y}\left(\right)$ denotes the $\mathsf{\epsilon }$-greedy strategy, where the agent selects the random action in the action space $\mathbf{A}$ with the probability $\mathsf{\epsilon }$ and selects the action that maximizes the Q-function, as shown in Equation (37) with the probability $(1-\mathsf{\epsilon })$. The function $\mathrm{K}\mathrm{F}\left(\right)$ is the KF equations, which are similar to the predication and update equations in Algorithm 1. The output ${\widehat{\mathit{Q}}}_{ck}$ of the algorithm is exploited in the transfer alignment KF.

Generally, the computational load of the QKF is related to the number of the parallel KFs in the algorithm. From previous works, it is known that only a few explorative KFs are sufficient to improve the filtering performance evidently. In this paper, from Figure 3 and Algorithm 2, only one explorative KF and one transfer alignment KF are contained in the QKF algorithm. Thus, the computational load of the QKF is about two times larger than the standard KF. It is easy to complete the computation of the QKF in the interval of the measurement update. The moderate increase in the computational load is affordable for the current onboard computers.

To ensure the efficiency of the QKF algorithm, an important aspect is to design the bound of the state space $\mathbf{S}$ appropriately. It is expected that a dynamic state space with the bound stretched automatically can be designed in future works. On the basis of the QKF presented in Algorithm 2, the measurement innovation sequence in a time window could be taken into account to suppress the unfavorable effect of the measurement noise [40]. Although the process noise covariance matrix obtained from the algorithm may not be globally optimal, it is often an effective and simple approach to improve the filtering performance.

4. CRLB of Transfer Alignment System

The feasibility of the inter-spacecraft rapid transfer alignment method based on the attitude plus angular rate matching scheme is analyzed through the CRLB. The CRLB is a theoretical bound on the achievable state estimation accuracy for certain system models. It facilitates the potential performance analysis of the transfer alignment method before the numerical simulation of the filtering algorithm. For the linear discrete-time stochastic system formulated in Equations (26) and (30), the calculation process of the CRLB is described in Algorithm 3.

Algorithm 3: Calculation of CRLB

1: ${\mathit{J}}_0\leftarrow {\mathit{P}}_0^{-1}$

2: for k = 1, 2, …, K, do

3: ${\mathit{J}}_k\leftarrow {({\mathit{F}}_k{\mathit{J}}_{k-1}^{-1}{\mathit{F}}_k^{\mathrm{T}}+{\mathit{Q}}_k)}^{-1}$

4: if the measurement ${\mathit{y}}_k$ is available, then

5:  ${\mathit{J}}_k\leftarrow {\mathit{J}}_k+{\mathit{H}}_k^{\mathrm{T}}{\mathit{R}}_k^{-1}{\mathit{H}}_k$

6: end if

7: ${\widehat{\mathit{P}}}_k\leftarrow {\mathit{J}}_k^{-1}$

8: end for

9: return $\left\{{\widehat{\mathit{P}}}_k\right\}$

In the algorithm, ${\mathit{J}}_k$ is the fisher information matrix calculated with the considered system model. The square roots of the diagonal elements of the calculated matrix ${\widehat{\mathit{P}}}_k$ provide the theoretical bound of the state estimation root mean square (RMS) error.

The CRLB analysis is implemented under the following conditions. The master spacecraft is an Earth satellite with an orbit altitude of 700 km and its attitude keeps in orientation to the Earth. The slave spacecraft is mounted on the master spacecraft. For the ADS on the master spacecraft, the attitude determination accuracy is 3″ and the angular rate determination accuracy is 0.02°/h. For the GU on the slave spacecraft, the ARW and RRW coefficients are 4 × 10⁻⁴°/h^0.5 and 1 × 10⁻³°/h^1.5 respectively. The scale factor error parameter vector is set as ${\left[\mathrm{500\; 500\; 500}\right]}^{\mathrm{T}}\mathrm{p}\mathrm{p}\mathrm{m}$ (parts per million) and the misalignment parameter vector is set as ${\left[\begin{array}{cc}\begin{array}{ccc}50″& 50″& 50″\end{array}& \begin{array}{ccc}50″& 50″& 50″\end{array}\end{array}\right]}^{\mathrm{T}}$. The update rate of the filter is 1 Hz. The total simulation time of the transfer alignment is 3600 s. The calibration maneuver is the sequential rotation around the three orthogonal axes of the master spacecraft body frame. Generally, the transfer alignment performance can be improved when the calibration maneuver angular rate is increased. Considering that the dynamic measurement performance of the typical star sensor may be degraded if its angular rate is larger than 2°/s, the rotation angular rate of the master spacecraft is set as 1°/s.

The CRLB for the inter-spacecraft rapid transfer alignment system described in Section 2 is calculated using Algorithm 3. Figure 4, Figure 5 and Figure 6 give the theoretical error bounds of the calibration parameters when the time length of the rotation around each axis is 100 s, 200 s, 400 s and 600 s, respectively.

From the CRLB curves, it is evident that the rapidity of the transfer alignment is guaranteed when a shorter rotation time is adopted. According to the analysis results, in the following numerical simulation, the time length of the rotation around each axis is set as 100 s for the calibration maneuver.

5. Simulation Results

To illustrate the high performance of the inter-spacecraft rapid transfer alignment method based on the QKF, the simulation results are shown in this section. For the data generation of the sensors, the simulation conditions are same as those in Section 4. In the transfer alignment KF, the initial state estimation error covariance matrix is set as follows.

$${\mathit{P}}_0=\left[\begin{array}{ccc}{p}_a^2{\mathbf{I}}_{3\times 3}& & \\ & p_r^2{\mathbf{I}}_{3\times 3}& \\ & & p_c^2{\mathbf{I}}_{3\times 3}\end{array}\right],$$

(41)

where $p_a=0.05°$, $p_r=0.05°/\mathrm{h}$. The elements in $p_c$ are larger than triple the magnitude of the calibration parameters given in Section 4. Similarly to the attitude determination KF [21], in the process noise covariance matrix ${\mathit{Q}}_k$, the sub-matrices ${\mathit{Q}}_{ak}$ and ${\mathit{Q}}_{rk}$ are designed according to the ARW and RRW coefficients of the GU. The sub-matrix ${\mathit{Q}}_{ck}$ is set as ${\mathit{Q}}_{ck}=q_c^2{\mathbf{I}}_{9\times 9}$, where the magnitude of the parameter $q_c$ is ${10}^{-5}$. In the measurement noise covariance matrix ${\mathit{R}}_k$, the sub-matrices are set as ${\mathit{R}}_{\rho k}=r_{\rho }^2{\mathbf{I}}_{3\times 3}$ and ${\mathit{R}}_{\omega k}=r_{\omega }^2{\mathbf{I}}_{3\times 3}$, where $r_{\rho }=3″$ and $r_{\omega }=0.02°/\mathrm{h}$. For the inter-spacecraft rapid transfer alignment, the initial attitude estimate ${\widehat{\mathit{q}}}_0$ is obtained from the ADS on the master spacecraft, and the state vector related to the calibration parameters is initialized as zero.

The first simulation is performed to illustrate the high performance of the attitude plus angular rate matching scheme presented in Section 2. The presented attitude plus angular rate matching scheme is compared with the traditional attitude matching scheme via the simulation. The average RMS errors of the attitude plus angular rate matching scheme and the attitude matching scheme for the estimation of the attitude and the calibration parameters obtained from 10 individual trials are plotted together in Figure 7, Figure 8 and Figure 9.

It is seen from Figure 7, Figure 8 and Figure 9 that the presented scheme performs better than the traditional scheme. The reason is that more measurement information is available for the transfer alignment in the attitude plus angular rate matching scheme. With the simulation results shown in the figures above, we conclude that the presented attitude plus angular rate matching scheme is efficient for the inter-spacecraft transfer alignment.

The aim of the second simulation is to illustrate the performance of the QKF presented in Section 3. In the QKF algorithm, the Q-learning parameters are set as $\mathsf{\alpha }=0.2$, $\mathsf{\gamma }=0.8$ and $\mathsf{\epsilon }=0.5$. Generally, the parameters can be designed with a trial-and-error method via the numerical simulation. For the considered noise covariance adaptation problem in the filtering algorithm, it was found in previous works [39,44] that the influence of the design parameters is not significant when they are chosen in certain scopes. In the pre-determined set related to the state space $\mathbf{S}$ for the tuning of the process noise covariance matrix, the sub-matrix ${\mathit{Q}}_{ck}^{\left(s\right)}$ is set as

$${\mathit{Q}}_{ck}^{\left(s\right)}={\left({\lambda }^{\left(s\right)}\right)}^2{q}_c^2{\mathbf{I}}_{9\times 9},$$

(42)

where ${\lambda }^{\left(s\right)}$ is a scalar factor in the range of ${\lambda }^{\left(s\right)}\in [1, 100]$, with 11 different values inside this interval. The cardinality of the pre-determined set is rather small, which is beneficial for the Q-learning approach in order to achieve a reasonable result. In fact, for the considered scenario, a small parameter set is effective to improve the filtering performance. Nevertheless, a sophisticated Q-learning approach may be required for more complicated problems.

The estimation error curves of the attitude and the calibration parameters as well as their corresponding $\pm 3\mathsf{\sigma }$ error bounds computed from the filter’s error covariance matrix are shown in Figure 10, Figure 11 and Figure 12.

It is seen from the figures that the estimation errors of the calibration parameters diminish rapidly after the calibration maneuver is performed. Both the scale factor and the misalignment estimation errors converge in about 600 s. All the estimation error curves of the QKF are contained in the corresponding error bounds, which indicate the consistency of the filtering algorithm. The simulation result illustrates that all the systematic errors of the GU on the slave spacecraft in the system model can be calibrated rapidly and accurately with the ADS on the master spacecraft. Simultaneously, it shows that the QKF is feasible for the inter-spacecraft rapid transfer alignment.

To facilitate the comparison, the average RMS errors of different transfer alignment methods are listed in

Table 1. Average RMS error of state estimation error for transfer alignment.

Matching Scheme	Filtering Algorithm	Average RMS Error
		Attitude (″)	Scale Factor (ppm)	Misalignment (″)
Attitude	KF	0.78	140.17	23.14
Attitude + angular rate	KF	0.42	113.40	13.91
Attitude + angular rate	QKF	0.31	69.80	6.30

.

Obviously, the presented attitude plus angular rate matching scheme outperforms the traditional attitude matching scheme. The state estimation accuracy can be improved when the presented QKF is used instead of the standard KF.

Furthermore, to demonstrate the advantage of the QKF algorithm, the simulation of the inter-spacecraft rapid transfer alignment methods based on different filtering algorithms are performed, including the standard KF, the Sage-Husa AKF and the QKF. For a fair comparison, three filtering algorithms share the basic filtering parameters, including the initialization parameters, the state transition matrix, the measurement matrix and the initial noise covariance matrices. The average RMS errors of the KF, the AKF and the QKF for the estimation of the attitude and the calibration parameters obtained from 10 individual trials are plotted in Figure 13, Figure 14 and Figure 15.

The above figures illustrate that the QKF obtains the highest state estimation accuracy in comparison with the KF and the AKF. This result is distinguishable in terms of both the attitude and the calibration parameters estimation. It indicates that the QKF is more effective than the AKF to identify the appropriate process noise covariance matrix and consequently improves the filtering performance.

To illustrate the performance of the presented QKF algorithm in different scenarios, the numerical simulation is implemented with different measurement noise levels. In the inter-spacecraft rapid transfer alignment system, when the attitude measurement noise standard deviation increases from 1″ to 10″, the average attitude estimation RMS errors of the KF and the QKF obtained from 10 individual trials are plotted in Figure 16. It indicates that the QKF algorithm is less sensitive to the variation in the measurement noise levels.

To sum up, it is apparent from the simulation results that the presented inter-spacecraft rapid transfer alignment method is valuable to satisfy the rapidity and accuracy requirements for the on-orbit calibration. The presented attitude plus angular rate matching scheme performs better than the traditional one. The QKF has considerable potential for practical applications in space missions as both the KF algorithm and the Q-learning approach are familiar for aerospace engineers. The basic principle of the presented method can be further applied to calibrate other attitude sensors, although the system model should be modified separately depending on the specific application.

6. Conclusions

The accurate and rapid transfer alignment is critical for the GU on the slave spacecraft, which influences the performance of the slave spacecraft after it is released from the master spacecraft. To satisfy the requirements of the space missions, this paper presents the novel inter-spacecraft rapid transfer alignment method based on the attitude plus angular rate matching scheme. For fine tuning the process noise covariance matrix of the transfer alignment KF, the Q-learning approach is incorporated with the model-based KF, which results in the QKF algorithm. Simulations are implemented for the performance evaluation. The simulation results demonstrate that the presented method can estimate all the calibration parameters in the transfer alignment model, and the misalignment estimation accuracy of less than 2 mrad is achievable within 10 min. The attitude plus angular rate matching scheme presents improved performance compared with the traditional attitude matching scheme. The QKF achieves the best estimation accuracy compared with the KF and AKF algorithms. Further work is required to evaluate the efficiency of the presented inter-spacecraft rapid transfer alignment method in practical applications and optimize the algorithm design according to the real error behavior. Although demonstrated for space missions, the presented method can be elaborated for information fusion and error calibration in other platforms for future research.