Long-Periodic Analysis of Boresight Misalignment of Ziyuan3-01 Three-Line Camera

The Ziyuan3-01 (ZY3-01) satellite is China’s first civilian stereo surveying and mapping satellite to meet the 1:50,000 scale mapping requirements, and has been operated in orbit for 10 years. The boresight misalignment of the three-line camera (TLC) is an essential factor affecting the geolocation accuracy, which is a principal concern for stereo mapping satellites. However, the relative relationships of TLC are often regarded as fixed for the same ground scene in most traditional geometric calibrations, without considering the on-orbit long-periodic changes. In this paper, we propose a long-periodic method to analyze and estimate the boresight misalignments between three cameras, with the attitude estimation of a nadir (NAD) camera as the benchmark. Offsets and drifts of the three cameras were calculated and calibrated with different compensation models using scale invariant feature transform (SIFT) points as the ground control. Ten simultaneous NAD–Forward (FWD)–Backward (BWD) imagery of the ZY3-01 satellite acquired from 2012 to 2020 were selected to verify the long-periodic changes in TLC boresight misalignments. The results indicate that the boresight alignment angles of ZY3-01 TLC are dynamic during the long-periodic flight, but the structure of TLC is stable for the misalignments of both FWD and BWD within only 7 arc seconds, which can provide a positive reference for subsequent satellite design and long-periodic on-orbit geometric calibration.


Introduction
Stereo surveying and mapping satellites have been widely used in Earth and planet observation for their excellent performance in acquiring stereo images [1][2][3][4]. High-precision topographical mapping can be derived from the satellite stereo imagery of the same target area from different view directions [5]. In the geometric processing of high-resolution satellite images, establishing a rigorous geometric model is a key step. Exterior orientation parameters (EOPs), interior orientation parameters (IOPs) and mounting parameters of cameras are required in direct georeferencing using a rigorous geometric model [6][7][8]. The dynamic EOPs can be obtained with attitude determination and positioning devices, such as GPS receivers, star trackers and gyroscopes, while the IOPs and mounting parameters are accurately calibrated in the laboratory before launch. However, camera parameters calibrated in the laboratory often change due to some unavoidable factors, such as space provide better service for on-orbit geometry calibration and thus improve the quality of geolocation. Moreover, long-periodic geometric accuracy analysis and prediction are also needed to study and verify without ground control measurements. In this study, a method with long-periodic analysis of boresight misalignment of TLC is proposed to estimate and validate the stability of ZY3-01 satellite TLC. Ten simultaneous NAD-FWD-BWD imagery of the ZY3-01 satellite acquired from 2012 to 2020 were selected to build the time series and verify the characteristics and variations of long-periodic TLC boresight misalignments using SIFT points as the ground control points (GCPs).
The remainder of this paper is organized as follows. In Section 2, the rigorous imaging geometric model of the TLC and the analysis approach of long-periodic TLC boresight misalignment are introduced. Experiments and corresponding results are described in Section 3. Then, camera stability, error sources and the effect of long time series are discussed in Section 4. Finally, Section 5 presents the conclusions.

Imaging Geometric Model of TLC
The ZY3-01 satellite operates in a sun-synchronous orbit with an orbital altitude of about 505.98 km, the semi-major axis and eccentricity of which are about 6876.98 km and 0, respectively. The orbital inclination is about 97.42 • and the orbit recursive period is about 59 days [31]. As shown in Figure 1, the TLC consists of three linear CCD arrays on the focal plane of the optical system, which are arranged in parallel and perpendicular to the flight direction. During the satellite flight, the three independent cameras image the ground target from different observation angles with a synchronous scanning period, so as to obtain the FWD, NAD and BWD images. The rigorous imaging geometric model of the TLC is as follows.
where X g Y g Z g T ECEF represents the ground coordinates under the Earth-centered, Earth-fixed (ECEF) coordinate frame; t F , t N and t B are the imaging time of the FWD, NAD and BWD cameras, respectively; T ECEF is the camera position; m F , m N and m B are the scale factors of the TLC; and R ECEF nav (t) is the transformation matrix from the navigation coordinate system to the ECEF coordinate frame, which is calculated using the attitude model.
are the interior orientation elements, representing the CCD detector look angles corresponding to the image point in the body coordinates for FWD, NAD and BWD cameras, respectively.

Geometric Calibration of the TLC Boresight Misalignment
According to Equation (1), the model errors mainly come from satellite positioning deviations, attitude errors and camera placement angle errors. Moreover, the influences on geometric accuracy from satellite position deviations and attitude error are basically the same because of the high correlation of model parameters, so the satellite position deviations can be equivalent to attitude errors [36]. Pitch, roll and yaw are used to describe the deviations between the corresponding axes of camera coordinates and satellite body coordinates. Then, the boresight calibration matrices in Equation (1) can be described as follows.
This indicates that there are errors in both R ECEF nav (t N ) and R N in Formula (1) for the NAD camera model, and the same for FWD and BWD cameras. During the imaging, the real-time positions of the satellite are obtained by a GPS receiver, while the attitude parameters are determined by star trackers and a gyroscope. High-precision GCPs are used in the geometric calibration of ZY3-01TLC. However, the attitude errors of pitch (δϕ(t)), roll (δω(t)) and yaw (δκ(t)) cannot be separated from the mounting angle error using this point-based calibration model. That means, whereR F ,R N ,R B are the estimated matrices of the mounting angles for FWD, NAD and BWD cameras, respectively. δR(t F ), δR(t N ) and δR(t B ) are the corresponding matrices of attitude error corrections.
Attitude errors should be eliminated before calibrating the TLC boresight misalignment. For short-time imaging, linear models as follows are used to estimate the attitude errors of the pitch, roll and yaw angles at the imaging time t.
where ϕ 0 , ω 0 , κ 0 represent the offsets and ϕ 1 , ω 1 , κ 1 represent the drifts of three angles varying over time. Considering the integrated structure of three cameras, the attitude errors can eliminate the attitude errors, and can also counteract the boresight misalignment of the NAD camera for FWD and BWD images acquired at the same imaging time t. In other words, the relative misalignment between cameras can be estimated. Here, a nadir camera is used as the benchmark for its nadir looking angle and its higher resolution compared with the FWD and BWD cameras. Even the attitude errors of pitch, roll and yaw cannot be separated from the mounting angle errors for each independent camera; it can be inferred that the errors estimated from GCPs are mainly caused by the boresight misalignment between the FWD and NAD cameras, or the BWD and NAD cameras. On this basis, Equation (1) can be rewritten as follows.
where δR NF is the error matrix of the boresight misalignment between the FWD camera and the NAD camera, and δR NB is that between the NAD camera and BWD camera, respectively. According to Formula (2), the three mounting angles of boresight misalignment can be derived from the following equations.
where δϕ NF , δω NF and δκ NF are the correction values of the angles between the FWD camera and NAD the camera, and δϕ NB , δω NB and δκ NB are those of the angles between the BWD camera and the NAD camera. Since the angles are very small, Equation (7) and Equation (8) can be simplified as follows.
where V is the correction to the observation's errors, A is the coefficient matrix according to Equations (5) and (6) and L is the matrix of the point observations. These values are solved by the least squares method. GCPs are extracted by the SIFT algorithm [39], which maintains invariance to rotation, scale scaling and brightness changes to a certain extent. The image and the Gaussian function are convolved to build scale space of the image with different Gaussians.
where L(x, y, σ) represents the image scale space, G(x, y, σ) is the Gaussian kernel function at scale σ and I(x, y) represents the original scale pixels. Key points can be determined by extreme value detection of the extracted feature points on different Gaussian scales; then, the feature descriptor with 128 dimensions can be generated. The key points are determined as follows.
where m(x, y) and θ(x, y) are the amplitude and phase of gradient at the position (x, y), and L(x, y) is the pixel value at a corresponding scale.

Long-Periodic Analysis of Boresight Misalignment
Time series of related parameters, including ϕ 0 , ω 0 , κ 0 , ϕ 1 , ω 1 , κ 1 and the values of RMSEs of sample and line directions in the image space, are built to analyze and evaluate the changes in boresight misalignments during the ten years of operation of the ZY3-01 satellite. The values of average, variance and cross-covariance of time series are mainly used to describe the variation characteristics of boresight misalignments, and thus reflect the stability of cameras in the long-time operation. Assuming that the time series of one boresight misalignment parameter Z t is {Z t1 , Z t2 , ..., Z tn }, t1, t2, . . . , tn represent n observation time nodes. Then, the average µ t , variance σ 2 and cross-covariance γ of Z t are as follows.
It should be noted that the boresight analysis in this study is based on the geometric calibrations of the ZY3-01 satellite (see the method in references [13,15]). The iterative method is used to solve the interior orientations. Such errors are not the focus of this study and are not covered here. The approach of boresight misalignment calibration for ZY3-01 TLC is shown in Figure 2. First, the attitude errors of the NAD camera are calculated by Equation (4) using the GCPs, EOPs and IOPs calibrated in the laboratory before the satellite's launch. The attitude errors of pitch, roll and yaw angles represent the misalignment of the NAD camera in the body coordinate. Second, the FWD and BWD images at the same imaging time with the NAD camera are selected. GCPs in the FWD and BWD coverage are used to estimate the errors in FWD and BWD cameras. Then, boresight misalignment between the FWD and NAD cameras, and between the BWD and NAD cameras are calculated based on Equations (5)- (11). After the least squares adjustment, the misalignment of the FWD and BWD cameras can be accurately restored relative to the NAD camera. At last, time series of related parameters for boresight misalignments are analyzed and evaluated using Equation (14).

Experimental Datasets
In this study, 10 ZY3-01 datasets acquired from 2012 to 2020 were selected to verify the changes and calibrations of TLC boresight misalignment, as listed in Table 1. Each dataset has FWD, NAD and BWD images, which were imaged simultaneously by the three cameras. The time spans of each dataset are about 5 s. As shown in Figure 3, the spatial coverage of these images is in the range of 113.5 • -119.5 • E and 32.5 • -38.5 • N, where the terrain is relatively flat with the elevation varying from 10 m to 450 m above sea level. The DEM and Digital Orthophoto Map (DOM) generated from ZY3-01 stereo images were used as the reference data. The GSD of the DOMs and DEMs are 2 m and 15 m, respectively. The planimetric accuracy of the DOMs is about 2 m, while the vertical accuracy of the DEMs is about 3 m. Here, the DOM is taken as the base image, and 13,581 SIFT points are extracted to be the virtual GCPs. After that, the object-space coordinates of these GCPs were measured manually from the reference DOMs and DEMs. First, the object-space coordinates of these points are measured from the reference DOMs and DEMs. Then, the image-space coordinates of these points are measured in the corresponding TLC image. According to the procedure of attitude error estimation, the boresight misalignment of the NAD camera is first performed, the results of which would be the reference to analyze the changes in the mounting angles between the navigation coordinate and the NAD camera coordinate. Then, the simultaneous FWD and BWD cameras are evaluated with GCPs in the corresponding FWD and BWD coverage. After that, the boresight misalignments of FWD and BWD cameras relative to the NAD camera would be estimated and compensated by the mounting angle parameters.

Calibration of NAD Camera
It is more rigorous to separately calculate the boresight changes of each camera in a unified reference. Considering the complexity of calculation led by the coupling relationship between attitude errors and boresight changes, the NAD camera is selected as the benchmark for a more robust solution. It should be pointed out that the IOPs of all datasets are normalized to exclude their influence, and the mounting angle calibrated in October 2012 is taken as the initial matrix. Table 2 shows the detailed results of ϕ 0 , ω 0 , κ 0 , ϕ 1 , ω 1 and κ 1 of the NAD camera in Equation (4) calculated using the GCPs. Figure 3 shows the curves of ϕ 0 , ω 0 and κ 0 angles changing over time. It can be seen from Table 2 and Figure 4 that the mounting angles of the NAD camera change significantly over time. The angles of roll and yaw vary more dramatically than pitch angle, and the maximum change of the roll angle reaches 74.13 arc seconds. The most significant changes appeared during the period from 2014 to 2015, where all three angles showed great fluctuation because the restart of the satellite temperature control system led by a solar panel fault on one side of the satellite occurred in October 2015.    In addition, the average angular velocity of the yaw angle is 0.23 arc seconds/s, which is much larger than that of the pitch and the roll angles, while the averages of the latter two are 0.04 arc seconds/s and −0.01 arc seconds/s, respectively. The maximum angular velocity of the yaw angle reaches 0.87 arc seconds/s, which is also larger than that of pitch and roll angles. This means the yaw angle of the ZY3-01 satellite has more uncertainties than the other two angles. Figure 5 shows the cross-covariances of time series among ϕ 0 , ω 0 and κ 0 angles. All the peaks occur at lag 0, which means that ϕ 0 , ω 0 and κ 0 angles have the greatest correlation with themselves. However, the curves with similar oscillation are not close to 0, which also indicates that there are cross-correlations between the three time series. With the boresight misalignment of the NAD camera above, the mean orientation errors of all three cameras in image space are calculated using ground check points. Table 3 shows the accuracies of the Sample and Line directions in the image space of the TLC. From the results, the orientation accuracy of the NAD camera is 0.6 pixels in the Sample direction and 0.39 pixels in the Line direction, which reached the sub-pixel level after compensation with the boresight misalignment in Table 2. However, the averages of Sample and Line of the FWD camera are 1.27 pixels and 2.11 pixels, respectively, and 1.90 pixels and 3.39 pixels for the BWD camera. Both the FWD and BWD cameras have significant orientation deviations in the along-track and cross-track directions, and those of the BWD camera are especially obvious. This also indicates that the angles between the three cameras are not fixed.

Boresight Misalignment Calibration and Characteristics of FWD and BWD Cameras
Taking the estimated mounting angle matrix of the NAD camera as the reference, the boresight misalignments of the FWD and BWD cameras are further calibrated. Since the misalignment between the navigation coordinates and the camera coordinates is timedependent, the offsets and drifts of three angles for the FWD and BWD cameras are estimated and calibrated consistent with the NAD camera using Equation (4). It is represented by six specific parameters, which are (δϕ NF0 , δϕ NF1 , δω NF0 , δω NF1 , δκ NF0 , δκ NF1 ) for the FWD camera, and (δϕ NB0 , δϕ NB1 , δω NB0 , δω NB1 , δκ NB0 , δκ NB1 ) for the BWD camera. Considering that the angles between the three cameras are relatively stable and the yaw angle of the ZY3-01 satellite has more uncertainty than the other two angles, a model with four parameters, (δϕ NF0 ,δϕ NF1 ,δω NF0 ,δω NF1 ) and (δϕ NB0 ,δϕ NB1 ,δω NB0 ,δω NB1 ), would be selected to estimate the misalignments and accuracy in image space. Then, ignoring the time variation parameters, a model with only two parameters, (δϕ NF0 , δω NF0 ) and (δϕ NB0 , δω NB0 ), are set to estimate the misalignment and accuracy in image space. Tables 4 and 5 list the statistical accuracies in image space before and after the compensation of misalignment for FWD and BWD cameras, respectively. The root mean square errors (RMSEs) in the image space of both FWD and BWD cameras are significantly reduced, reaching the sub-pixel level. The accuracy of the BWD camera is improved more obviously for the average RMSEs in the Sample and Line directions, which are less than 0.5 pixels with six parameters, while the average RMSEs were 1.90 and 3.39 pixels, respectively, before the compensation. After the compensation, the accuracies of the FWD and BWD cameras are almost the same.   Figure 6 displays the variations in RMSEs in the image Sample and Line directions before and after compensation for the FWD and BWD cameras. It can be seen that the accuracies using four parameters are similar to those using six parameters, especially in the Sample direction for both FWD and BWD cameras. The RMSEs in the image space of the FWD camera using two time-independent parameters are slightly larger than those using four or six parameters. This also indicates that the misalignment of the TLC is also associated with time parameters, but the effect is not significant. According to the estimations of FWD and BWD cameras using two parameters, the boresight misalignments of FWD and BWD cameras are calculated using the method described in Section 2.2. Figures 7 and 8 show the angle changes and normalized autocovariance sequences of FWD and BWD cameras with the NAD camera as the benchmark. The misalignments of both FWD and BWD cameras are within 7 arc seconds, which indicates that the structure of the TLC is relatively stable. Moreover, the misalignment of the BWD camera is larger than that of the FWD camera. Pitch and roll angles between the BWD camera and the NAD camera present some gradual accumulation over time, while the angles between the FWD camera and the NAD camera change randomly. From Figure 8, the randomness of the pitch angles between the FWD and NAD cameras is more obvious than others, while the roll angles between the FWD and NAD cameras show some correlation over time. Since the curves decline on both sides in Figure 8c,d, the autocovariance sequences of pitch and roll angles between the BWD and NAD cameras are similar, which is consistent with the gradual accumulation in Figure 7.

Camera stability from Boresight Misalignment Calibration
The TLC of the ZY3-01 satellite is relatively stable overall, as the installation deviation of the whole TLC and the attitude measurement system varies by tens of arc seconds (See Figure 4), and the boresight misalignments of FWD and BWD cameras are both within 7 arc seconds during its 10 years of operation. These results can provide reference for subsequent satellite design and long-time on-orbit calibration. Moreover, attitude errors are often the main errors compared with errors of alignment angles [29,38]. Combined with the changes in the mounting angles of the NAD camera in Figure 4 and the variations in boresight angles between ZY3-01's two star trackers shown in Figure 9, it can be seen that all errors above 10 arc seconds are caused by attitude errors. However, the positioning accuracies of FWD and BWD cameras in the image space are still large when only one compensation matrix of the NAD camera is applied for three cameras. The RMSE in the Sample direction is more than 1 pixel, while the RMSE in the Line direction is more than 2 pixels or even 3 pixels. That means the alignment angles of ZY3-01 TLC are dynamic during the operation. The reason for this misalignment may be the influences of the temperature, radiation or other environmental factors, which are also worthy of investigation and would be helpful for camera stability. In addition, the three angles of boresight misalignments of TLC are also different. The shifts of pitch are relatively stable and smaller than the other two, and the drifts of pitch and roll angles are almost the same from the change curves. The drastic changes in roll and yaw parameters maybe caused by inaccurate control points or narrow FoV. In general, the pitch parameter affects the accuracy in the Line direction, while roll and yaw affect the Sample direction. When the values of roll and yaw parameters are small, they are correlated. Moreover, the yaw angle of the ZY3-01 satellite has more uncertainties than the other two angles. Considering that the yaw angle introduces a very small error in the image space for very narrow FoV images [32], the compensation model with four parameters omitted the yaw angle. The results also show that the results of the four-parameter model and the six-parameter model are almost the same.

Error Sources for Boresight Misalignment Calibration
Imaging geometric model of TLC denotes the geometric relationship between the payload and the satellite measurement equipment. The factors, including time synchronization, attitude, orbit measurements, on-orbit equipment installation and camera distortions, are the main error sources for stereo mapping of the ZY3-01 satellite. The boresight misalignments of TLC will affect the geolocation accuracy. There are deviations between the attitudes determined by the navigation coordinates system and the real camera attitudes. These deviations can be eliminated through removing the misalignment between the camera and the navigation coordinates by geometric calibration or by offset compensation. The attitude deviations are highly intercorrelated for single-camera satellites such as agile satellites, while the attitude deviations of TLC can be further divided into the misalignments between the navigation coordinates and the satellite body coordinates, and the deviations between TLC coordinates and the satellite body coordinates. In other words, both R ECEF nav (t) and R N , R F , R B in the geometric model are related to misalignments of TLC and require error corrections. However, the geometric calibration of the ZY3-01 satellite is carried out step by step based on the internal and external elements. For cameras, the internal calibration mainly focuses on the focal length of camera and the look angle of every detector on CCD arrays, which reflect the internal geometric relationship of the camera. Their accuracy mainly depends on the distortion errors of cameras. The external calibration is concerned with changes in attitudes and mounting matrix of cameras, and the georeferencing accuracy is mainly influenced by on-orbit equipment installation errors. It is difficult to calibrate each separate parameter because of their strong correlations. In this study, errors of cameras are all absorbed by the boresight misalignments caused by the installation, which is why the three FWD-NAD-BWD images simultaneously captured are used to ensure as consistent attitude parameters as possible in the body coordinates. The misalignments of the three cameras estimated here reflect the relative relationship between FWD and NAD cameras, and FWD and NAD cameras. The detailed misalignment for each individual camera needs further study. Moreover, the misalignment difference between FWD and BWD shown in Figures 6 and 7 also indicate that the factors influencing the boresight misalignment of each camera are not exactly the same. The remaining errors should be further considered with internal calibration.
In addition, the accuracy of ground points is also an important factor affecting boresight misalignment calibration of TLC. From Equation (1), the accuracy of the GCP measurements should be improved to ensure the direct georeferencing accuracy. However, such a number of high-precision points are difficult to obtain in the case of long time series image coverage. The SIFT points used as the ground points in this study can be acquired easily and can guarantee the distribution in the coverage area. Moreover, the close accuracies of these points ensure the consistency of all ground points in the misalignment calibration, and therefore, the effect on three cameras can be seen as same as one another. Thus, they do not change the relative relationship of the three cameras. However, high-precision ground points such as observations from Real-Time Kinematic (RTK) would improve the georeferencing accuracy. If the geometric calibration is performed in the same area, then the accuracy of ground points will be closely related to the boresight misalignment calibration.

Effects of Long Time Series
Different compensation models for TLC are required for further high-precision geolocation or mapping. Geometric boresight calibration is needed for improving the geometric accuracy of imagery because of the errors. As the time interval of TLC is 28 s for the ZY3-01 satellite imaging the same ground target, the compensation models of boresight misalignments are different for the same ground scene. Even in the case of simultaneous imaging, the same boresight misalignment calibration is not suitable for the three cameras for improving the accuracy. The shifts of the BWD camera accumulate over time but those of the FWD camera do not, which means that the misalignments of three cameras are different. Therefore, more validations combined with long time series and short-term calibrations are needed for further study. Moreover, orientation parameters R ECEF nav (t) and R N , R F , R B in the geometric model are also time-dependent; these instantaneous and longterm characteristics of TLC would directly affect the accuracy of positioning. Because the installation matrices of three cameras often change during the flight, the calibration in orbit should consider the instantaneous and long-term changes, which form the dynamical characters and differences of TLC. The advantages and necessity of long time series analysis deserve more attention.

Conclusions
This study presents a long-periodic method to analyze and calculate the boresight misalignments between three cameras, with the attitude estimation of a nadir (NAD) camera as the benchmark. Experiments using ten simultaneous NAD-FWD-BWD imagery of the ZY3-01 satellite acquired from 2012 to 2020 were performed to estimate the offsets and drifts of the three cameras and thus the relative position between the three cameras with the alignment angles. Time series of related parameters for boresight misalignments were built and analyzed. Our conclusions are as follows.
(1) The structure of ZY3-01 TLC was stable overall in the 10 years of operation, and the alignment angles of TLC were dynamic over time.
(2) Compensation models of TLC are different for each camera. High-precision geopositioning or mapping should consider the differences and relations of each camera rather than only a unified installation matrix. (3) Both the camera coordinates and the navigation coordinates change significantly with time. Therefore, regular geometric calibration is necessary for improving the positioning accuracy of high-resolution satellites. (4) Long-periodic analyses of TLC boresight misalignments indicate the changes in TLC angles, but the change patterns need further investigation.