Long-Periodic Analysis of Boresight Misalignment of Ziyuan3-01 Three-Line Camera

Abstract

The Ziyuan3-01 (ZY3-01) satellite is China’s first civilian stereo surveying and mapping satellite to meet the 1:50,000 scale mapping requirements, and has been operated in orbit for 10 years. The boresight misalignment of the three-line camera (TLC) is an essential factor affecting the geolocation accuracy, which is a principal concern for stereo mapping satellites. However, the relative relationships of TLC are often regarded as fixed for the same ground scene in most traditional geometric calibrations, without considering the on-orbit long-periodic changes. In this paper, we propose a long-periodic method to analyze and estimate the boresight misalignments between three cameras, with the attitude estimation of a nadir (NAD) camera as the benchmark. Offsets and drifts of the three cameras were calculated and calibrated with different compensation models using scale invariant feature transform (SIFT) points as the ground control. Ten simultaneous NAD–Forward (FWD)–Backward (BWD) imagery of the ZY3-01 satellite acquired from 2012 to 2020 were selected to verify the long-periodic changes in TLC boresight misalignments. The results indicate that the boresight alignment angles of ZY3-01 TLC are dynamic during the long-periodic flight, but the structure of TLC is stable for the misalignments of both FWD and BWD within only 7 arc seconds, which can provide a positive reference for subsequent satellite design and long-periodic on-orbit geometric calibration.

1. Introduction

Stereo surveying and mapping satellites have been widely used in Earth and planet observation for their excellent performance in acquiring stereo images [1,2,3,4]. High-precision topographical mapping can be derived from the satellite stereo imagery of the same target area from different view directions [5]. In the geometric processing of high-resolution satellite images, establishing a rigorous geometric model is a key step. Exterior orientation parameters (EOPs), interior orientation parameters (IOPs) and mounting parameters of cameras are required in direct georeferencing using a rigorous geometric model [6,7,8]. The dynamic EOPs can be obtained with attitude determination and positioning devices, such as GPS receivers, star trackers and gyroscopes, while the IOPs and mounting parameters are accurately calibrated in the laboratory before launch. However, camera parameters calibrated in the laboratory often change due to some unavoidable factors, such as space environment variation, especially temperature changes [9,10], equipment aging, geometric alignment of cameras and many other factors [11,12,13,14,15,16].

In order to guarantee the geometric accuracy of the geospatial information products, on-orbit geometric calibration of cameras is an essential step to obtain accurate interior orientation parameters and mounting parameters, which directly determine the geo-positioning accuracy. Many studies have conducted research on geometric calibration and analysis of cameras for currently available satellite sensors [2,6,17,18,19,20,21,22,23,24], such as Ikonos, OrbView-3, SPOT-5, IRS-P6, MISR, ALOS/PRISM, MOMS-2P and HRSC of Mars [2,24], and China’s Tianhui, Gaofen and ZY-3 [25,26,27,28,29]. These approaches mainly improve the image georeferencing accuracy and thus the mapping product accuracy. For example, the tilting angles of camera boresight and IOPs are calibrated using large stereo image blocks; the geo-positioning accuracy of Ikonos images has reached about 8 m with ground control information [4,17]. The direct georeferencing accuracy of IRS-P6 images was improved from about 2000 to 250 m after calibration of the individual sensor alignment, the inter-camera alignment and the focal plane [19]. The geo-positioning accuracy was improved to 15 m with geometric camera calibration [18]. Calibrations such as block adjustments or image self-calibration are also used to improve the georeferencing accuracy for ALOS/PRISM images [30] or the quality of the Digital Elevation Model (DEM) for the SPOT 5 satellite [6].

As the first domestic stereo mapping optical satellite in China, the ZY3-01 satellite was launched in January 2012 and equipped with a multispectral camera (four bands: blue, green, red and near infrared) and panchromatic TLC (spectral range: 0.5–0.88 um) [31]. The FWD and BWD cameras are installed at inclinations of ±22° from the NAD camera with a base-to-height ratio of 0.87. The spatial resolution is 3.5 m for the FWD and BWD cameras, and 2.1 m for the NAD camera. These satellite images are widely used for mapping and production of 1:50,000 scale cartographic products. There have been several geometric calibrations of cameras to assess, validate and improve the geometric positioning accuracy [7,8,11,12,13,14,32,33,34,35,36]. Tang et al. (2012, [29]) proposed an image geometry model of the ZY3-01 satellite using virtual CCD line-array imaging. Zhang et al. (2014, [14]) proposed a bias compensation model of TLC for each virtual strip scene. Cao (2016, [8]) used a line-based geometric model of ZY3-01 TLC to improve the direct georeferencing accuracy, and a better georeferencing accuracy of about 3.0 m in planimetry and 2.5 m in height was achieved. Similarly, the nonlinear bias compensation model with cubic splines [32], the compensation model with the shift and drift of both pitch and roll angle [33] and the integrated geometric self-calibration of stereo cameras [34] for ZY3-01 TLC provide satisfactory compensation or calibration for the TLC. Moreover, after these geometric calibrations, the direct geo-positioning accuracy can be significantly improved with GCPs, line features or other extracted features as the control information. High-resolution linear array satellite images have an extremely small field of view (FoV), resulting in high correlation between attitude errors and mounting angle errors, or interior errors [15,16]. Thus, most of these methods distinguish the attitude errors and mounting angles errors of TLC less, or ignore the dynamic changes in the relative positions of the cameras. There are also some studies that consider the relative differences between cameras. MISR compensates the attitude of each camera separately, but does not address the mounting angles of cameras [1]. MODIS has studied the geometric accuracy changes in 36 years caused by the effective focal length of camera, while the attitudes and installation structure are not considered [37]. Pan et al. (2016, [33]) validate that the relationships among TLCs are dynamic during imaging for the same ground scene, and compensation for the TLC as a unit introduces a height error of about 1 m for the ZY3-01 satellite. However, the long-periodic dynamic changes of TLC on orbit during the operation are less studied, especially for the ZY3-01 satellite.

The ZY3-01 satellite has been operated in orbit for 10 years. An integrated and stable structure of the star trackers and TLC was adopted to reduce the relative pointing changes between them [38]. Long-periodic analyses of boresight misalignment for TLC are needed to model the geometric characteristics of sensors during the long-term on-orbit operation, which can help precisely investigate the image error sources and propagation, and then provide better service for on-orbit geometry calibration and thus improve the quality of geolocation. Moreover, long-periodic geometric accuracy analysis and prediction are also needed to study and verify without ground control measurements. In this study, a method with long-periodic analysis of boresight misalignment of TLC is proposed to estimate and validate the stability of ZY3-01 satellite TLC. Ten simultaneous NAD–FWD–BWD imagery of the ZY3-01 satellite acquired from 2012 to 2020 were selected to build the time series and verify the characteristics and variations of long-periodic TLC boresight misalignments using SIFT points as the ground control points (GCPs).

The remainder of this paper is organized as follows. In Section 2, the rigorous imaging geometric model of the TLC and the analysis approach of long-periodic TLC boresight misalignment are introduced. Experiments and corresponding results are described in Section 3. Then, camera stability, error sources and the effect of long time series are discussed in Section 4. Finally, Section 5 presents the conclusions.

2. Methodology

2.1. Imaging Geometric Model of TLC

The ZY3-01 satellite operates in a sun-synchronous orbit with an orbital altitude of about 505.98 km, the semi-major axis and eccentricity of which are about 6876.98 km and 0, respectively. The orbital inclination is about 97.42° and the orbit recursive period is about 59 days [31]. As shown in Figure 1, the TLC consists of three linear CCD arrays on the focal plane of the optical system, which are arranged in parallel and perpendicular to the flight direction. During the satellite flight, the three independent cameras image the ground target from different observation angles with a synchronous scanning period, so as to obtain the FWD, NAD and BWD images. The rigorous imaging geometric model of the TLC is as follows.

$$\begin{gathered} {\left[\begin{array}{c}{X}_g\\ Y_g\\ Z_g\end{array}\right]}_{ECEF}={\left[\begin{array}{c}{X}_S\left(t_F\right)\\ Y_S\left(t_F\right)\\ Z_S\left(t_F\right)\end{array}\right]}_{ECEF}+m_F\cdot {\mathit{R}}_{nav}^{ECEF}\left(t_F\right)\cdot {\mathit{R}}_F\cdot \left[\begin{array}{c}tan\left({\psi }_{Fy}\left(c_F\right)\right)\\ -tan\left({\psi }_{Fx}\left(c_F\right)\right)\\ 1\end{array}\right] \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}={\left[\begin{array}{c}{X}_S\left(t_N\right)\\ Y_S\left(t_N\right)\\ Z_S\left(t_N\right)\end{array}\right]}_{ECEF}+m_N\cdot {\mathit{R}}_{nav}^{ECEF}\left(t_N\right)\cdot {\mathit{R}}_N\cdot \left[\begin{array}{c}tan\left({\psi }_{Ny}\left(c_N\right)\right)\\ -tan\left({\psi }_{Nx}\left(c_N\right)\right)\\ 1\end{array}\right] \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}={\left[\begin{array}{c}{X}_S\left(t_B\right)\\ Y_S\left(t_B\right)\\ Z_S\left(t_B\right)\end{array}\right]}_{ECEF}+m_B\cdot {\mathit{R}}_{nav}^{ECEF}\left(t_B\right)\cdot {\mathit{R}}_B\cdot \left[\begin{array}{c}tan\left({\psi }_{By}\left(c_B\right)\right)\\ -tan\left({\psi }_{Bx}\left(c_B\right)\right)\\ 1\end{array}\right] \end{gathered}$$

(1)

where ${\left[\begin{array}{ccc}{X}_g& Y_g& Z_g\end{array}\right]}_{ECEF}^T$ represents the ground coordinates under the Earth-centered, Earth-fixed (ECEF) coordinate frame; $t_F$, $t_N$ and $t_B$ are the imaging time of the FWD, NAD and BWD cameras, respectively; ${\left[\begin{array}{ccc}{X}_S\left(t\right)& Y_S\left(t\right)& Z_S\left(t\right)\end{array}\right]}_{ECEF}^T$ is the camera position; $m_F$, $m_N$ and $m_B$ are the scale factors of the TLC; and ${\mathit{R}}_{nav}^{ECEF}\left(t\right)$ is the transformation matrix from the navigation coordinate system to the ECEF coordinate frame, which is calculated using the attitude model. ${\mathit{R}}_F,{\mathit{R}}_N,{\mathit{R}}_B$ are the boresight calibration matrices. $\left({\psi }_{Fy}\left(c_F\right),{\psi }_{Fx}\left(c_F\right)\right)$,$\left({\psi }_{Ny}\left(c_N\right),{\psi }_{Nx}\left(c_N\right)\right)$ and$\left({\psi }_{By}\left(c_B\right),{\psi }_{Bx}\left(c_B\right)\right)$ are the interior orientation elements, representing the CCD detector look angles corresponding to the image point in the body coordinates for FWD, NAD and BWD cameras, respectively.

2.2. Geometric Calibration of the TLC Boresight Misalignment

According to Equation (1), the model errors mainly come from satellite positioning deviations, attitude errors and camera placement angle errors. Moreover, the influences on geometric accuracy from satellite position deviations and attitude error are basically the same because of the high correlation of model parameters, so the satellite position deviations can be equivalent to attitude errors [36]. Pitch, roll and yaw are used to describe the deviations between the corresponding axes of camera coordinates and satellite body coordinates. Then, the boresight calibration matrices in Equation (1) can be described as follows.

$${\mathit{R}}_N={\mathit{R}}_{pitch}{\mathit{R}}_{roll}{\mathit{R}}_{yaw}$$

(2)

This indicates that there are errors in both ${\mathit{R}}_{nav}^{ECEF}\left(t_N\right)$ and ${\mathit{R}}_N$ in Formula (1) for the NAD camera model, and the same for FWD and BWD cameras. During the imaging, the real-time positions of the satellite are obtained by a GPS receiver, while the attitude parameters are determined by star trackers and a gyroscope. High-precision GCPs are used in the geometric calibration of ZY3-01TLC. However, the attitude errors of pitch ($\delta \phi \left(t\right)$), roll ($\delta \omega \left(t\right)$) and yaw ($\delta \kappa \left(t\right)$) cannot be separated from the mounting angle error using this point-based calibration model. That means,

$$\left\{\begin{array}{c}{\widehat{\mathit{R}}}_F=\delta \mathit{R}\left(t_F\right)\cdot {\mathit{R}}_F\\ {\widehat{\mathit{R}}}_N=\delta \mathit{R}\left(t_N\right)\cdot {\mathit{R}}_N\\ {\widehat{\mathit{R}}}_B=\delta \mathit{R}\left(t_B\right)\cdot {\mathit{R}}_B\end{array}\right.$$

(3)

where ${\widehat{\mathit{R}}}_F,{\widehat{\mathit{R}}}_N,{\widehat{\mathit{R}}}_B$ are the estimated matrices of the mounting angles for FWD, NAD and BWD cameras, respectively. $\delta \mathit{R}\left(t_F\right)$, $\delta \mathit{R}\left(t_N\right)$ and $\delta \mathit{R}\left(t_B\right)$ are the corresponding matrices of attitude error corrections.

Attitude errors should be eliminated before calibrating the TLC boresight misalignment. For short-time imaging, linear models as follows are used to estimate the attitude errors of the pitch, roll and yaw angles at the imaging time t.

$$\left\{\begin{array}{c}\delta \phi \left(t\right)={\phi }_0+{\phi }_1\cdot t\\ \delta \omega \left(t\right)={\omega }_0+{\omega }_1\cdot t\\ \delta \kappa \left(t\right)={\kappa }_0+{\kappa }_1\cdot t\end{array}\right.$$

(4)

where ${\phi }_0, {\omega }_0, {\kappa }_0$ represent the offsets and ${\phi }_1, {\omega }_1, {\kappa }_1$ represent the drifts of three angles varying over time. Considering the integrated structure of three cameras, the attitude errors can eliminate the attitude errors, and can also counteract the boresight misalignment of the NAD camera for FWD and BWD images acquired at the same imaging time t. In other words, the relative misalignment between cameras can be estimated. Here, a nadir camera is used as the benchmark for its nadir looking angle and its higher resolution compared with the FWD and BWD cameras. Even the attitude errors of pitch, roll and yaw cannot be separated from the mounting angle errors for each independent camera; it can be inferred that the errors estimated from GCPs are mainly caused by the boresight misalignment between the FWD and NAD cameras, or the BWD and NAD cameras. On this basis, Equation (1) can be rewritten as follows.

$${\left[\begin{array}{c}{X}_{Fg}\\ Y_{Fg}\\ Z_{Fg}\end{array}\right]}_{ECEF}={\left[\begin{array}{c}{X}_S\left(t\right)\\ Y_S\left(t\right)\\ Z_S\left(t\right)\end{array}\right]}_{ECEF}+m_F\cdot {\mathit{R}}_{nav}^{ECEF}\left(t\right)\cdot \delta {\mathit{R}}_{NF}\cdot {\mathit{R}}_F\cdot \left[\begin{array}{c}tan\left({\psi }_{Fy}\left(c_F\right)\right)\\ -tan\left({\psi }_{Fx}\left(c_F\right)\right)\\ 1\end{array}\right]$$

(5)

$${\left[\begin{array}{c}{X}_{Bg}\\ Y_{Bg}\\ Z_{Bg}\end{array}\right]}_{ECEF}={\left[\begin{array}{c}{X}_S\left(t\right)\\ Y_S\left(t\right)\\ Z_S\left(t\right)\end{array}\right]}_{ECEF}+m_B\cdot {\mathit{R}}_{nav}^{ECEF}\left(t\right)\cdot \delta {\mathit{R}}_{NB}\cdot {\mathit{R}}_B\cdot \left[\begin{array}{c}tan\left({\psi }_{By}\left(c_B\right)\right)\\ -tan\left({\psi }_{Bx}\left(c_B\right)\right)\\ 1\end{array}\right]$$

(6)

where $\delta {\mathit{R}}_{NF}$ is the error matrix of the boresight misalignment between the FWD camera and the NAD camera, and $\delta {\mathit{R}}_{NB}$ is that between the NAD camera and BWD camera, respectively.

According to Formula (2), the three mounting angles of boresight misalignment can be derived from the following equations.

$$\delta {\mathit{R}}_{NF}=\delta {\mathit{R}}_{{\phi }_{NF}}\cdot \delta {\mathit{R}}_{{\omega }_{NF}}\cdot \delta {\mathit{R}}_{{\kappa }_{NF}}=\left[\begin{array}{ccc}cos\delta {\phi }_{NF}& 0& sin\delta {\phi }_{NF}\\ 0& 1& 0\\ -sin\delta {\phi }_{NF}& 0& cos\delta {\phi }_{NF}\end{array}\right]\cdot \left[\begin{array}{ccc}1& 0& 0\\ 0& cos\delta {\omega }_{NF}& -sin\delta {\omega }_{NF}\\ 0& sin\delta {\omega }_{NF}& cos\delta {\omega }_{NF}\end{array}\right]\cdot \left[\begin{array}{ccc}cos\delta {\kappa }_{NF}& -sin\delta {\kappa }_{NF}& 0\\ sin\delta {\kappa }_{NF}& cos\delta {\kappa }_{NF}& 0\\ 0& 0& 1\end{array}\right]$$

(7)

$$\delta {\mathit{R}}_{NB}=\delta {\mathit{R}}_{{\phi }_{NB}}\cdot \delta {\mathit{R}}_{{\omega }_{NB}}\cdot \delta {\mathit{R}}_{{\kappa }_{NB}}=\left[\begin{array}{ccc}cos\delta {\phi }_{NB}& 0& sin\delta {\phi }_{NB}\\ 0& 1& 0\\ -sin\delta {\phi }_{NB}& 0& cos\delta {\phi }_{NB}\end{array}\right]\cdot \left[\begin{array}{ccc}1& 0& 0\\ 0& cos\delta {\omega }_{NB}& -sin\delta {\omega }_{NB}\\ 0& sin\delta {\omega }_{NB}& cos\delta {\omega }_{NB}\end{array}\right]\cdot \left[\begin{array}{ccc}cos\delta {\kappa }_{NB}& -sin\delta {\kappa }_{NB}& 0\\ sin\delta {\kappa }_{NB}& cos\delta {\kappa }_{NB}& 0\\ 0& 0& 1\end{array}\right]$$

(8)

where $\delta {\phi }_{NF}$, $\delta {\omega }_{NF}$ and $\delta {\kappa }_{NF}$ are the correction values of the angles between the FWD camera and NAD the camera, and $\delta {\phi }_{NB}$, $\delta {\omega }_{NB}$ and $\delta {\kappa }_{NB}$ are those of the angles between the BWD camera and the NAD camera. Since the angles are very small, Equation (7) and Equation (8) can be simplified as follows.

$$\delta {\mathit{R}}_{NF}=\left[\begin{array}{ccc}1& -\delta {\kappa }_{NF}& -\delta {\phi }_{NF}\\ \delta {\kappa }_{NF}& 1& -\delta {\omega }_{NF}\\ \delta {\phi }_{NF}& \delta {\omega }_{NF}& 1\end{array}\right]$$

(9)

$$\delta {\mathit{R}}_{NB}=\left[\begin{array}{ccc}1& -\delta {\kappa }_{NB}& -\delta {\phi }_{NB}\\ \delta {\kappa }_{NB}& 1& -\delta {\omega }_{NB}\\ \delta {\phi }_{NB}& \delta {\omega }_{NB}& 1\end{array}\right]$$

(10)

Taking $\mathit{X}={[\delta {\phi }_{NF},\delta {\omega }_{NF},\delta {\kappa }_{NF}]}^T$ and $\mathit{X}={\left[\delta {\phi }_{NB},\delta {\omega }_{NB},\delta {\kappa }_{NB}\right]}^T$ as unknowns, Equations (9) and (10) are substituted into Equations (5) and (6), respectively. Then, using GCPs in each camera coverage, the error equations can be established as follows.

$$\mathit{V}=\mathit{A}\mathit{X}-\mathit{L}$$

(11)

where V is the correction to the observation’s errors, A is the coefficient matrix according to Equations (5) and (6) and L is the matrix of the point observations. These values are solved by the least squares method.

GCPs are extracted by the SIFT algorithm [39], which maintains invariance to rotation, scale scaling and brightness changes to a certain extent. The image and the Gaussian function are convolved to build scale space of the image with different Gaussians.

$$L\left(x,y,\sigma \right)=G\left(x,y,\sigma \right)*I\left(x,y\right)$$

(12)

where $L\left(x,y,\sigma \right)$ represents the image scale space, $G\left(x,y,\sigma \right)$ is the Gaussian kernel function at scale $\sigma$ and $I\left(x,y\right)$ represents the original scale pixels. Key points can be determined by extreme value detection of the extracted feature points on different Gaussian scales; then, the feature descriptor with 128 dimensions can be generated. The key points are determined as follows.

$$\begin{array}{c}m\left(x,y\right)=\sqrt{{\left(L\left(x+1,y\right)-L\left(x-1,y\right)\right)}^2+{\left(L\left(x,y+1\right)-L\left(x,y-1\right)\right)}^2}\\ \theta \left(x,y\right)=ta{n}^{-1}\left(L\left(x,y+1\right)-L\left(x,y-1\right)\right)/\left(L\left(x+1,y\right)-L\left(x-1,y\right)\right)\end{array}$$

(13)

where $m\left(x,y\right)$ and $\theta \left(x,y\right)$ are the amplitude and phase of gradient at the position$\left(x,y\right), \mathrm{and} L\left(x,y\right)$ is the pixel value at a corresponding scale.

2.3. Long-Periodic Analysis of Boresight Misalignment

Time series of related parameters, including${\phi }_0, {\omega }_0, {\kappa }_0,{\phi }_1, {\omega }_1, {\kappa }_1$ and the values of RMSEs of sample and line directions in the image space, are built to analyze and evaluate the changes in boresight misalignments during the ten years of operation of the ZY3-01 satellite. The values of average, variance and cross-covariance of time series are mainly used to describe the variation characteristics of boresight misalignments, and thus reflect the stability of cameras in the long-time operation. Assuming that the time series of one boresight misalignment parameter $Z_t$is $\{Z_{t1},Z_{t2}, ..., Z_{tn}\}$, $t1,t2, \dots , tn$ represent n observation time nodes. Then, the average ${\mu }_t$, variance ${\sigma }^2$and cross-covariance $\gamma$ of $Z_t$ are as follows.

$$\left\{\begin{array}{c}{\mathit{\mu }}_{\mathit{t}}=\mathit{E}\left({\mathit{Z}}_{\mathit{t}}\right), \mathit{t}=t1,t2, \dots , tn\\ {\mathit{\sigma }}^2=\frac{\sum {\left({\mathit{Z}}_{\mathit{t}}-{\mathit{\mu }}_{\mathit{t}}\right)}^2}{n}\\ \mathit{\gamma }=\mathit{C}\mathit{o}\mathit{v}({\mathit{Z}}_{\mathit{t}},{\mathit{Z}}_{\mathit{s}})=\mathit{E}\left[\left({\mathit{Z}}_{\mathit{t}}-{\mu }_t\right)({\mathit{Z}}_{\mathit{s}}-{\mathit{\mu }}_{\mathit{s}}\right)], \mathit{t}, \mathit{s}=t1,t2, \dots , tn\end{array}\right.$$

(14)

It should be noted that the boresight analysis in this study is based on the geometric calibrations of the ZY3-01 satellite (see the method in references [13,15]). The iterative method is used to solve the interior orientations. Such errors are not the focus of this study and are not covered here. The approach of boresight misalignment calibration for ZY3-01 TLC is shown in Figure 2. First, the attitude errors of the NAD camera are calculated by Equation (4) using the GCPs, EOPs and IOPs calibrated in the laboratory before the satellite’s launch. The attitude errors of pitch, roll and yaw angles represent the misalignment of the NAD camera in the body coordinate. Second, the FWD and BWD images at the same imaging time with the NAD camera are selected. GCPs in the FWD and BWD coverage are used to estimate the errors in FWD and BWD cameras. Then, boresight misalignment between the FWD and NAD cameras, and between the BWD and NAD cameras are calculated based on Equations (5)–(11). After the least squares adjustment, the misalignment of the FWD and BWD cameras can be accurately restored relative to the NAD camera. At last, time series of related parameters for boresight misalignments are analyzed and evaluated using Equation (14).

3. Results

3.1. Experimental Datasets

In this study, 10 ZY3-01 datasets acquired from 2012 to 2020 were selected to verify the changes and calibrations of TLC boresight misalignment, as listed in

Table 1. Spatial coverage of the images acquired by the TLC of satellite ZY3-01.

Datasets	FWD		NAD		BWD
20121001	113.831° E	33.852° N	114.353° E	35.838° N	114.898° E	37.807° N
20130604	114.401° E	34.110° N	114.916° E	36.061° N	115.472° E	38.069° N
20140220	117.181° E	33.480° N	117.705° E	35.462° N	118.261° E	37.435° N
20151206	116.728° E	37.268° N	117.272° E	39.224° N	117.856° E	41.214° N
20160921	115.018° E	34.101° N	115.534° E	36.061° N	116.093° E	38.059° N
20170510	114.368° E	34.101° N	114.876° E	36.060° N	115.421° E	38.059° N
20171029	114.430° E	34.102° N	114.950° E	36.061° N	115.508° E	38.060° N
20180904	115.516° E	34.102° N	116.035° E	36.061° N	116.591° E	38.059° N
20190701	115.764° E	34.102° N	116.281° E	36.061° N	116.835° E	38.059° N
20200521	118.121° E	32.913° N	118.625° E	34.874° N	119.165° E	36.873° N

. Each dataset has FWD, NAD and BWD images, which were imaged simultaneously by the three cameras. The time spans of each dataset are about 5 s. As shown in Figure 3, the spatial coverage of these images is in the range of 113.5°–119.5° E and 32.5°–38.5° N, where the terrain is relatively flat with the elevation varying from 10 m to 450 m above sea level. The DEM and Digital Orthophoto Map (DOM) generated from ZY3-01 stereo images were used as the reference data. The GSD of the DOMs and DEMs are 2 m and 15 m, respectively. The planimetric accuracy of the DOMs is about 2 m, while the vertical accuracy of the DEMs is about 3 m. Here, the DOM is taken as the base image, and 13,581 SIFT points are extracted to be the virtual GCPs. After that, the object–space coordinates of these GCPs were measured manually from the reference DOMs and DEMs. First, the object–space coordinates of these points are measured from the reference DOMs and DEMs. Then, the image–space coordinates of these points are measured in the corresponding TLC image. According to the procedure of attitude error estimation, the boresight misalignment of the NAD camera is first performed, the results of which would be the reference to analyze the changes in the mounting angles between the navigation coordinate and the NAD camera coordinate. Then, the simultaneous FWD and BWD cameras are evaluated with GCPs in the corresponding FWD and BWD coverage. After that, the boresight misalignments of FWD and BWD cameras relative to the NAD camera would be estimated and compensated by the mounting angle parameters.

3.2. Calibration of NAD Camera

It is more rigorous to separately calculate the boresight changes of each camera in a unified reference. Considering the complexity of calculation led by the coupling relationship between attitude errors and boresight changes, the NAD camera is selected as the benchmark for a more robust solution. It should be pointed out that the IOPs of all datasets are normalized to exclude their influence, and the mounting angle calibrated in October 2012 is taken as the initial matrix.

Table 2. The boresight alignment of the NAD camera of the ZY3-01 satellite.

Datasets	${\mathit{\phi }}_0$	${\mathit{\omega }}_0$	${\mathit{\kappa }}_0$	${\mathit{\phi }}_1$	${\mathit{\omega }}_1$	${\mathit{\kappa }}_1$
	(Arc Seconds)			(Arc Seconds/s)
20121001	0.00	0.00	0.00	0.16	−0.12	0.02
20130604	−2.59	−1.64	6.25	−0.05	0.03	−0.40
20140220	−4.34	−7.99	−2.38	−0.17	−0.12	−0.87
20151206	−21.94	−68.81	−41.16	0.04	0.04	0.33
20160921	−26.02	−74.13	−31.04	0.06	0.05	0.76
20170510	−24.70	−62.63	−26.39	−0.04	−0.07	0.62
20171029	−21.85	−63.96	−32.30	0.11	−0.06	0.48
20180904	−27.23	−60.05	−27.32	−0.03	−0.02	0.17
20190701	−25.66	−22.96	−12.22	0.24	0.21	0.86
20200521	−21.20	−27.87	−16.66	0.06	−0.05	0.36
Average	−17.55	−39.00	−18.32	0.04	−0.01	0.23
Absolute Maximum	27.23	74.13	41.16	0.24	0.21	0.87
Variance	115.53	889.64	250.93	0.014	0.0098	0.29

shows the detailed results of ${\phi }_0$, ${\omega }_0$, ${\kappa }_0$, ${\phi }_1$, ${\omega }_{1 }$and ${\kappa }_1$of the NAD camera in Equation (4) calculated using the GCPs. Figure 3 shows the curves of ${\phi }_0$, ${\omega }_0$ and ${\kappa }_0$ angles changing over time. It can be seen from Table 2 and Figure 4 that the mounting angles of the NAD camera change significantly over time. The angles of roll and yaw vary more dramatically than pitch angle, and the maximum change of the roll angle reaches 74.13 arc seconds. The most significant changes appeared during the period from 2014 to 2015, where all three angles showed great fluctuation because the restart of the satellite temperature control system led by a solar panel fault on one side of the satellite occurred in October 2015.

In addition, the average angular velocity of the yaw angle is 0.23 arc seconds/s, which is much larger than that of the pitch and the roll angles, while the averages of the latter two are 0.04 arc seconds/s and −0.01 arc seconds/s, respectively. The maximum angular velocity of the yaw angle reaches 0.87 arc seconds/s, which is also larger than that of pitch and roll angles. This means the yaw angle of the ZY3-01 satellite has more uncertainties than the other two angles. Figure 5 shows the cross-covariances of time series among ${\phi }_0$, ${\omega }_0$ and ${\kappa }_0$ angles. All the peaks occur at lag 0, which means that ${\phi }_0$, ${\omega }_0$ and ${\kappa }_0$ angles have the greatest correlation with themselves. However, the curves with similar oscillation are not close to 0, which also indicates that there are cross-correlations between the three time series.

With the boresight misalignment of the NAD camera above, the mean orientation errors of all three cameras in image space are calculated using ground check points.

Table 3. The RMSEs in the image space of the TLC using only the mounting angle of the NAD camera (pixels).

Dataset	NAD		FWD		BWD
	Sample	Line	Sample	Line	Sample	Line
20121001	0.79	0.19	0.76	0.49	0.62	0.61
20130604	0.57	0.46	0.43	1.82	1.36	1.61
20140220	0.62	0.26	0.81	2.26	0.47	2.16
20151206	0.45	0.43	2.60	1.79	1.07	3.98
20160921	0.50	0.36	1.63	3.93	1.42	3.50
20170510	0.73	0.51	0.84	2.92	2.57	3.49
20171029	0.59	0.39	0.73	2.54	3.21	3.90
20180904	0.49	0.39	2.26	1.69	2.02	4.58
20190701	0.52	0.35	1.94	2.02	2.08	4.82
20200521	0.77	0.51	0.74	1.66	4.19	5.22
Average	0.60	0.39	1.27	2.11	1.90	3.39
Variance	0.015	0.01	0.58	0.82	1.37	2.21

shows the accuracies of the Sample and Line directions in the image space of the TLC. From the results, the orientation accuracy of the NAD camera is 0.6 pixels in the Sample direction and 0.39 pixels in the Line direction, which reached the sub-pixel level after compensation with the boresight misalignment in Table 2. However, the averages of Sample and Line of the FWD camera are 1.27 pixels and 2.11 pixels, respectively, and 1.90 pixels and 3.39 pixels for the BWD camera. Both the FWD and BWD cameras have significant orientation deviations in the along-track and cross-track directions, and those of the BWD camera are especially obvious. This also indicates that the angles between the three cameras are not fixed.

3.3. Boresight Misalignment Calibration and Characteristics of FWD and BWD Cameras

Taking the estimated mounting angle matrix of the NAD camera as the reference, the boresight misalignments of the FWD and BWD cameras are further calibrated. Since the misalignment between the navigation coordinates and the camera coordinates is time-dependent, the offsets and drifts of three angles for the FWD and BWD cameras are estimated and calibrated consistent with the NAD camera using Equation (4). It is represented by six specific parameters, which are $\left(\delta {\phi }_{NF}{}_0, \delta {\phi }_{NF}{}_1, \delta {\omega }_{NF}{}_0, \delta {\omega }_{NF}{}_1, \delta {\kappa }_{NF}{}_0, \delta {\kappa }_{NF}{}_1\right)$ for the FWD camera, and ($\delta {\phi }_{NB}{}_0$,$\delta {\phi }_{NB}{}_1$,$\delta {\omega }_{NB}{}_0$,$\delta {\omega }_{NB}{}_1$,$\delta {\kappa }_{NB}{}_0$, $\delta {\kappa }_{NB}{}_1$) for the BWD camera. Considering that the angles between the three cameras are relatively stable and the yaw angle of the ZY3-01 satellite has more uncertainty than the other two angles, a model with four parameters, ($\delta {\phi }_{NF}{}_0$,$\delta {\phi }_{NF}{}_1$,$\delta {\omega }_{NF}{}_0$,$\delta {\omega }_{NF}{}_1$) and ($\delta {\phi }_{NB}{}_0$,$\delta {\phi }_{NB}{}_1$,$\delta {\omega }_{NB}{}_0$,$\delta {\omega }_{NB}{}_1$), would be selected to estimate the misalignments and accuracy in image space. Then, ignoring the time variation parameters, a model with only two parameters, $\left(\delta {\phi }_{NF}{}_0,\delta {\omega }_{NF}{}_0\right)$ and $\left(\delta {\phi }_{NB}{}_0,\delta {\omega }_{NB}{}_0\right)$, are set to estimate the misalignment and accuracy in image space.

Table 4. The RMSEs in the image space of the FWD camera before and after compensation (unit: pixels).

Datasets	Before	Sample Direction			Before	Line Direction
		After Compensation				After Compensation
		6 par.	4 par.	2 par.		6 par.	4 par.	2 par.
20121001	0.76	0.40	0.40	0.46	0.49	0.40	0.50	0.50
20130604	0.43	0.32	0.32	0.62	1.82	0.56	0.59	0.59
20140220	0.81	0.27	0.27	0.31	2.26	0.46	0.46	0.52
20151206	2.60	0.58	0.58	0.60	1.79	0.55	0.57	0.58
20160921	1.63	0.52	0.52	0.52	3.93	0.44	0.47	0.47
20170510	0.84	0.59	0.59	0.60	2.92	0.60	0.60	0.68
20171029	0.73	0.63	0.63	0.64	2.54	0.42	0.43	0.45
20180904	2.26	0.55	0.55	0.55	1.69	0.42	0.45	0.45
20190701	1.94	0.55	0.55	0.56	2.02	0.50	0.53	0.56
20200521	0.74	0.59	0.59	0.60	1.66	0.54	0.56	0.56
Average	1.27	0.5	0.5	0.55	2.11	0.49	0.52	0.54

Note: par. means parameters.

and

Table 5. The RMSEs in the image space of the BWD camera before and after compensation (unit: pixels).

Datasets	Before	Sample Direction			Before	Line Direction
		After Compensation				After Compensation
		6 par.	4 par.	2 par.		6 par.	4 par.	2 par.
20121001	0.62	0.39	0.39	0.80	0.61	0.53	0.60	0.58
20130604	1.36	0.31	0.31	0.74	1.61	0.58	0.59	0.59
20140220	0.47	0.33	0.33	0.36	2.16	0.30	0.33	0.33
20151206	1.07	0.51	0.51	0.51	3.98	0.47	0.47	0.48
20160921	1.42	0.37	0.37	0.37	3.50	0.47	0.51	0.53
20170510	2.57	0.58	0.58	0.67	3.49	0.61	0.64	0.66
20171029	3.21	0.57	0.57	0.67	3.90	0.47	0.49	0.56
20180904	2.02	0.45	0.45	0.46	4.58	0.43	0.48	0.51
20190701	2.08	0.51	0.51	0.60	4.82	0.40	0.40	0.40
20200521	4.19	0.73	0.73	0.73	5.22	0.54	0.55	0.57
Average	1.90	0.48	0.48	0.59	3.39	0.48	0.51	0.52

Note: par. means parameters.

list the statistical accuracies in image space before and after the compensation of misalignment for FWD and BWD cameras, respectively. The root mean square errors (RMSEs) in the image space of both FWD and BWD cameras are significantly reduced, reaching the sub-pixel level. The accuracy of the BWD camera is improved more obviously for the average RMSEs in the Sample and Line directions, which are less than 0.5 pixels with six parameters, while the average RMSEs were 1.90 and 3.39 pixels, respectively, before the compensation. After the compensation, the accuracies of the FWD and BWD cameras are almost the same.

Figure 6 displays the variations in RMSEs in the image Sample and Line directions before and after compensation for the FWD and BWD cameras. It can be seen that the accuracies using four parameters are similar to those using six parameters, especially in the Sample direction for both FWD and BWD cameras. The RMSEs in the image space of the FWD camera using two time-independent parameters are slightly larger than those using four or six parameters. This also indicates that the misalignment of the TLC is also associated with time parameters, but the effect is not significant. According to the estimations of FWD and BWD cameras using two parameters, the boresight misalignments of FWD and BWD cameras are calculated using the method described in Section 2.2. Figure 7 and Figure 8 show the angle changes and normalized autocovariance sequences of FWD and BWD cameras with the NAD camera as the benchmark. The misalignments of both FWD and BWD cameras are within 7 arc seconds, which indicates that the structure of the TLC is relatively stable. Moreover, the misalignment of the BWD camera is larger than that of the FWD camera. Pitch and roll angles between the BWD camera and the NAD camera present some gradual accumulation over time, while the angles between the FWD camera and the NAD camera change randomly. From Figure 8, the randomness of the pitch angles between the FWD and NAD cameras is more obvious than others, while the roll angles between the FWD and NAD cameras show some correlation over time. Since the curves decline on both sides in Figure 8c,d, the autocovariance sequences of pitch and roll angles between the BWD and NAD cameras are similar, which is consistent with the gradual accumulation in Figure 7.

4. Discussion

4.1. Camera stability from Boresight Misalignment Calibration

The TLC of the ZY3-01 satellite is relatively stable overall, as the installation deviation of the whole TLC and the attitude measurement system varies by tens of arc seconds (See Figure 4), and the boresight misalignments of FWD and BWD cameras are both within 7 arc seconds during its 10 years of operation. These results can provide reference for subsequent satellite design and long-time on-orbit calibration. Moreover, attitude errors are often the main errors compared with errors of alignment angles [29,38]. Combined with the changes in the mounting angles of the NAD camera in Figure 4 and the variations in boresight angles between ZY3-01’s two star trackers shown in Figure 9, it can be seen that all errors above 10 arc seconds are caused by attitude errors. However, the positioning accuracies of FWD and BWD cameras in the image space are still large when only one compensation matrix of the NAD camera is applied for three cameras. The RMSE in the Sample direction is more than 1 pixel, while the RMSE in the Line direction is more than 2 pixels or even 3 pixels. That means the alignment angles of ZY3-01 TLC are dynamic during the operation. The reason for this misalignment may be the influences of the temperature, radiation or other environmental factors, which are also worthy of investigation and would be helpful for camera stability.

In addition, the three angles of boresight misalignments of TLC are also different. The shifts of pitch are relatively stable and smaller than the other two, and the drifts of pitch and roll angles are almost the same from the change curves. The drastic changes in roll and yaw parameters maybe caused by inaccurate control points or narrow FoV. In general, the pitch parameter affects the accuracy in the Line direction, while roll and yaw affect the Sample direction. When the values of roll and yaw parameters are small, they are correlated. Moreover, the yaw angle of the ZY3-01 satellite has more uncertainties than the other two angles. Considering that the yaw angle introduces a very small error in the image space for very narrow FoV images [32], the compensation model with four parameters omitted the yaw angle. The results also show that the results of the four-parameter model and the six-parameter model are almost the same.

4.2. Error Sources for Boresight Misalignment Calibration

Imaging geometric model of TLC denotes the geometric relationship between the payload and the satellite measurement equipment. The factors, including time synchronization, attitude, orbit measurements, on-orbit equipment installation and camera distortions, are the main error sources for stereo mapping of the ZY3-01 satellite. The boresight misalignments of TLC will affect the geolocation accuracy. There are deviations between the attitudes determined by the navigation coordinates system and the real camera attitudes. These deviations can be eliminated through removing the misalignment between the camera and the navigation coordinates by geometric calibration or by offset compensation. The attitude deviations are highly intercorrelated for single-camera satellites such as agile satellites, while the attitude deviations of TLC can be further divided into the misalignments between the navigation coordinates and the satellite body coordinates, and the deviations between TLC coordinates and the satellite body coordinates. In other words, both ${\mathit{R}}_{nav}^{ECEF}\left(t\right)$ and ${\mathit{R}}_N,{\mathit{R}}_F,{\mathit{R}}_B$ in the geometric model are related to misalignments of TLC and require error corrections.

However, the geometric calibration of the ZY3-01 satellite is carried out step by step based on the internal and external elements. For cameras, the internal calibration mainly focuses on the focal length of camera and the look angle of every detector on CCD arrays, which reflect the internal geometric relationship of the camera. Their accuracy mainly depends on the distortion errors of cameras. The external calibration is concerned with changes in attitudes and mounting matrix of cameras, and the georeferencing accuracy is mainly influenced by on-orbit equipment installation errors. It is difficult to calibrate each separate parameter because of their strong correlations. In this study, errors of cameras are all absorbed by the boresight misalignments caused by the installation, which is why the three FWD–NAD–BWD images simultaneously captured are used to ensure as consistent attitude parameters as possible in the body coordinates. The misalignments of the three cameras estimated here reflect the relative relationship between FWD and NAD cameras, and FWD and NAD cameras. The detailed misalignment for each individual camera needs further study. Moreover, the misalignment difference between FWD and BWD shown in Figure 6 and Figure 7 also indicate that the factors influencing the boresight misalignment of each camera are not exactly the same. The remaining errors should be further considered with internal calibration.

In addition, the accuracy of ground points is also an important factor affecting boresight misalignment calibration of TLC. From Equation (1), the accuracy of the GCP measurements should be improved to ensure the direct georeferencing accuracy. However, such a number of high-precision points are difficult to obtain in the case of long time series image coverage. The SIFT points used as the ground points in this study can be acquired easily and can guarantee the distribution in the coverage area. Moreover, the close accuracies of these points ensure the consistency of all ground points in the misalignment calibration, and therefore, the effect on three cameras can be seen as same as one another. Thus, they do not change the relative relationship of the three cameras. However, high-precision ground points such as observations from Real-Time Kinematic (RTK) would improve the georeferencing accuracy. If the geometric calibration is performed in the same area, then the accuracy of ground points will be closely related to the boresight misalignment calibration.

4.3. Effects of Long Time Series

Different compensation models for TLC are required for further high-precision geolocation or mapping. Geometric boresight calibration is needed for improving the geometric accuracy of imagery because of the errors. As the time interval of TLC is 28 s for the ZY3-01 satellite imaging the same ground target, the compensation models of boresight misalignments are different for the same ground scene. Even in the case of simultaneous imaging, the same boresight misalignment calibration is not suitable for the three cameras for improving the accuracy. The shifts of the BWD camera accumulate over time but those of the FWD camera do not, which means that the misalignments of three cameras are different. Therefore, more validations combined with long time series and short-term calibrations are needed for further study. Moreover, orientation parameters ${\mathit{R}}_{nav}^{ECEF}\left(t\right)$ and ${\mathit{R}}_N,{\mathit{R}}_F,{\mathit{R}}_B$ in the geometric model are also time-dependent; these instantaneous and long-term characteristics of TLC would directly affect the accuracy of positioning. Because the installation matrices of three cameras often change during the flight, the calibration in orbit should consider the instantaneous and long-term changes, which form the dynamical characters and differences of TLC. The advantages and necessity of long time series analysis deserve more attention.

5. Conclusions

This study presents a long-periodic method to analyze and calculate the boresight misalignments between three cameras, with the attitude estimation of a nadir (NAD) camera as the benchmark. Experiments using ten simultaneous NAD–FWD–BWD imagery of the ZY3-01 satellite acquired from 2012 to 2020 were performed to estimate the offsets and drifts of the three cameras and thus the relative position between the three cameras with the alignment angles. Time series of related parameters for boresight misalignments were built and analyzed. Our conclusions are as follows.

(1)

The structure of ZY3-01 TLC was stable overall in the 10 years of operation, and the alignment angles of TLC were dynamic over time.

(2)

Compensation models of TLC are different for each camera. High-precision geo-positioning or mapping should consider the differences and relations of each camera rather than only a unified installation matrix.

(3)

Both the camera coordinates and the navigation coordinates change significantly with time. Therefore, regular geometric calibration is necessary for improving the positioning accuracy of high-resolution satellites.

(4)

Long-periodic analyses of TLC boresight misalignments indicate the changes in TLC angles, but the change patterns need further investigation.