A New Combined Adjustment Model for Geolocation Accuracy Improvement of Multiple Sources Optical and SAR Imagery

Abstract

Numerous earth observation data obtained from different platforms have been widely used in various fields, and geometric calibration is a fundamental step for these applications. Traditional calibration methods are developed based on the rational function model (RFM), which is produced by image vendors as a substitution of the rigorous sensor model (RSM). Generally, the fitting accuracy of the RFM is much higher than 1 pixel, whereas the result decreases to several pixels in mountainous areas, especially for Synthetic Aperture Radar (SAR) imagery. Therefore, this paper proposes a new combined adjustment for geolocation accuracy improvement of multiple sources satellite SAR and optical imagery. Tie points are extracted based on a robust image matching algorithm, and relationships between the parameters of the range-doppler (RD) model and the RFM are developed by transformed into the same Geodetic Coordinate systems. At the same time, a heterogeneous weight strategy is designed for better convergence. Experimental results indicate that our proposed model can achieve much higher geolocation accuracy with approximately 2.60 pixels in the X direction and 3.50 pixels in the Y direction. Compared with traditional methods developed based on RFM, our proposed model provides a new way for synergistic use of multiple sources remote sensing data.

1. Introduction

With the development of satellite imaging technology, it is increasingly common to obtain repeated observations of the same object from multiple sources in a short time, which provides dozens of imagery widely used in many fields, such as 3D reconstruction [1], change detection [2] and semantic classification [3]. Nowadays, The application of multiple sources airborne and spaceborne remote sensing imagery is increasing popular in archaeological and cultural heritage as a supplement to traditional methods [4], which will provide sufficient texture information. Terrestrial results obtained by laser scanning suffer from high cost and missing data, whereas the combination of photogrammetry provides an affordable and practical approach for the production of 3D models. Compared with spaceborne imagery, the application of airborne remote sensing images is widespread due to its high resolution which will provide enough details of buildings. In 2014, Xu et al. proposed a methodology by intergrating laser scanning and image-based 3D reconstruction techniques for the the production of 3D models [5]. Meyer et al. investigated an optimized Unmanned Aerial Vehicles (UAV) system for the reconstruction of large scale cultural heritage sites [6]. A digital 3D model of Asinou Church in Cyprus is obtained using a consumer-level DJI platform equipped with a GoPro camera, and a 3D printer was used to create a physical model of the church [7]. Moreover, multispectral and hyperspectral remote sensing data also contribute a lot in the classification of material features of cultural heritage [8]. During the production of 3D models, all these datasets have to be rectified due to the low geometric performance [4]. Some open-source tools including the structure from motion (SfM) [9] and dense multi-view 3D reconstruction (DMVR) [10] are widely applied. However, most of them are designed for optical images, whereas the integration of SAR data is not well-considered for photogrammetric applications.

Generally, the geolocation accuracy of obtained satellite images varies greatly according to different satellite platforms and imaging principles. Hence, the geometric processing of remote sensing images is a fundamental step for further photogrammetric applications. Traditional methods for obtaining geometric calibrated satellite images are developed based on the rigorous sensor model (RSM) [11]. Usually, the establishment of the RSM requires orbit, altitude or other information of the on-orbit satellite platform. Therefore, the formula of the RSMs can be rather complicated.

As a replacement, generic models that fitting the RSM are proposed such as the direct linear transformation (DLT) model [12] and the rational function model (RFM) [13]. These kinds of generic models use polynomial functions to build the relationship between image-space and object-space coordinates. All of them are independent of the distinct characteristics of satellite sensors. Therefore, the RFM is widely used in photogrammetric processing of remote sensing images due to its simplicity of implementation and standardization [14]. Generally, the RFM is developed based on a third-order polynomial, and parameters of the RFM are recognized as rational polynomial coefficients (RPCs). Experiments have been conducted to verify the feasibility and efficiency of the RFM with various optical remote sensing datasets [15,16,17,18]. In 2010, Teo et al. compared three block adjustment models based on SPOT images [19]. Experimental results showed that the geometric performance of all three models are similar. With the help of ground control points (GCPs), all methods discussed in this paper can significantly improve the geolocation accuracy. Choi et al. also investigated the 3D performance of the vendor-provided RPCs using two stereo pairs of high-resolution GeoEye-1 and WorldView-2. The results indicated that the performance of the RFM and the RSM are nearly the same, and the plane accuracy without any GCPs can reach about 2.3 m [20].

Differently from traditional optical satellite images, satellite images obtained from Synthetic Aperture Radar (SAR) sensors can provide valuable information at all times and all weather [21]. Benefitting from all these characteristics, the technology of SAR has been greatly improved. World-class SAR sensors, such as the TerraSAR-X [22], the ALOS [23] and the COSMO- SkyMed [24], can provide images with an accuracy of higher than 10 m. The single look complex (SLC) images from the TerraSAR-X platform especially can even reach decimeter level. In contrast, the obtained performance of the Chinese GF-3 SAR imagery is relatively poor, with an accuracy of approximately 40 m in plane [25,26]. Therefore, investigations have been conducted to improve the geolocation accuracy of GF-3 SAR images [27,28,29,30]. Most of them are experimented based on the RPCs provided by image vendors. Differently, coefficients of the RD model are supplied for most world-class SAR sensors instead of the RFM. Furthermore, the production of RPCs by users will introduce extra fitting errors inevitable. Generally, RPCs can be approximated based on the terrain-independent method [31]. A set of virtual GCPs are arranged in a grid-shape format on planes located at different heights. Usually the fitting accuracy of approximated RPCs is better than 5% pixels, whereas errors increased to pixel-level in areas with undulating terrain [32].

Compared with traditional satellites, the geometric performance of imagery obtained from small satellites, a new kind of satellite with small volume and quick response, is unstable and usually poor due to the low measurement accuracy [15,33,34,35]. Traditional methods using existed reference data (such as GCPs, digital orthophoto map (DOM) or LiDAR data) to improve their geolocation accuracy [36,37,38,39]. In practice, the collection of these reference data requires considerable financial and human resources. Therefore, combined adjustment methods designed for multiple sources satellite imagery are investigated. In 2015, Jeong et al. investigated the performance of images from IKONOS, QuickBird and KOMPSAT-2 [40]. Redundant observations were involved for geolocation accuracy improvement of multiple sources satellite images [41]. In contrast, the integration of optical and SAR images are seldom investigated. Furthermore, most of the previous studies are experimented based on the RFM, which is not suitable for the geometric processing of multiple sources optical and SAR images in most cases.

In this paper, we propose a new and generic combined adjustment model designed for optical and SAR satellite images. When considering aerial remote sensing images, coefficients of the RFM for optical imagery should be produced first by users before the application of our proposed model. By introducing the relationships between coordinates defined in the Geodetic Coordinate System and Cartesian Coordinate System, parameters of the RD model are transformed into the same system with the RFM. Therefore the normal equations for the combined adjustment model are developed based on an image-space compensation model. A heterogeneous weight strategy is introduced for better convergence. With the help of a popular modified Least-Square method, the ill-conditioned problem can be solved efficiently.

The remainder of this paper is organized as follows—the basic principles of the RFM based combined adjustment model are introduced in Section 2. Our proposed combined adjustment model and the determination of the heterogeneous weight strategy is shown in Section 3. In Section 4, experimental results are shown to verify the efficiency of our proposed method using multiple optical and SAR images covering the Mount Song area. Conclusions and discussions are drawn in Section 5.

2. Basic Principle of the RFM

Generally, the RSM is composed of various on-orbit information of satellite platforms, which leads to a complicated form. Therefore, the RFM is proposed as a substitution of the RSM. Usually, the relationship between image-space coordinates and object-space coordinates are described by two polynomials as:

$$\begin{array}{cc}\hfill x& =\frac{Nu{m}_S}{De{n}_S}=\frac{[1PLH\cdots PLH]{[a_0{a}_1{a}_2{a}_3\cdots a_{19}]}^T}{[1PLH\cdots PLH]{[b_0{b}_1{b}_2{b}_3\cdots b_{19}]}^T}\hfill \\ \hfill y& =\frac{Nu{m}_L}{De{n}_L}=\frac{[1PLH\cdots PLH]{[c_0{c}_1{c}_2{c}_3\cdots c_{19}]}^T}{[1PLH\cdots PLH]{[d_0{d}_1{d}_2{d}_3\cdots d_{19}]}^T},\hfill \end{array}$$

(1)

where $(x,y)$ is the normalized image-space coordinate. $(P$, L, $H)$ denotes the normalized latitude, longitude, and height in object-space. $Nu{m}_S$, $De{n}_S$, $Nu{m}_L$ and $De{n}_L$ are third-order polynomials consisting of 80 coefficients marked as $a_i$, $b_i$, $c_i$ and $d_i$$(i=0,1,2,\cdots$, $19)$.

And the normalized coordinates can be obtained according to Equation (2).

$$\begin{array}{cc}\hfill s& =x·SAMP\_SCALE+SAMP\_OFF\hfill \\ \hfill l& =y·LINE\_SCALE+LINE\_OFF\hfill \\ \hfill \varphi & =P·LAT\_SCALE+LAT\_OFF\hfill \\ \hfill \lambda & =L·LONG\_SCALE+LONG\_OFF\hfill \\ \hfill h& =H·HEIGHT\_SCALE+HEIGHT\_OFF,\hfill \end{array}$$

(2)

where $(\varphi ,\lambda ,h)$ is the geodetic latitude, longitude and height calculated with different offset and scale factors, respectively; $(s,l)$ represents the image sample and line number in pixels with pixel $(0,0)$ is the top-left of the image.

$$\begin{array}{cc}\hfill F_r& =a_0+a_1·s+a_2·l-r\hfill \\ \hfill F_c& =b_0+b_1·s+b_2·l-c.\hfill \end{array}$$

(3)

The RFM can fit the RSM well in most cases. However, geolocation error still exists due to the low measurement accuracy of on-orbit sensors. A commonly used model for systematically error compensation is the affine transformation model. In Equation (3), r and c are extracted image-space coordinates; $a_0,a_1,a_2,b_0,b_1,$ and $b_2$ are designed affine transformation coefficients. Usually, systematic errors can be greatly eliminated based on a translation model with $a_0$ and $b_0$. Hence, normal equations based on the RFM can be simplified and described as follows [42]:

$$V=A·X_A+B·X_B-l,P,$$

(4)

where V is the residual vector, A and B are the designed coefficient matrices containing partial derivatives of the unknowns; $X_A$ and $X_B$ are the correction vectors of the affine transformation parameters and object-space coordinates, respectively. l are the vectors of residual errors and P denotes the designed weight matrix.

The normal equations can be established from Equation (4), according to the principle of least-squares adjustment:

$$\left[\begin{array}{ccc}{A}^{T}PA& & A^{T}PB\\ B^{T}PA& & B^{T}PB\end{array}\right]\left[\begin{array}{c}{X}_A\\ X_B\end{array}\right]=\left[\begin{array}{c}{A}^{T}Pl\\ B^{T}Pl.\end{array}\right]$$

(5)

3. Methodology

3.1. Overview

Generally, optical remote sensing imagery can be processed based on the RFM due to its simplicity and standardization. Instead, the geometric processing of SAR imagery is usually conducted based on the RD model due to the lack of RPCs. In this paper, a new combined adjustment model that aggregates the RD model and the RFM is proposed for geometric calibration of multiple sources optical and SAR imagery. The workflow is shown in Figure 1. Firstly, the sensor orientation of SAR images is processed for systematic error compensation, which can be considered as the coarse-calibration stage. Secondly, conjugate tie points are extracted using a feature-based OS-SIFT method, which is a more robust and efficient remote sensing image-matching method compared with others in computer vision [43]. Subsequently, the proposed combined adjustment model consisting of the RD model and RFM is applied to fulfill the fine-calibration stage. Therefore, the geolocation accuracy of calibrated SAR and optical images can be improved significantly, which facilitates further photogrammetric applications.

3.2. Sensor Orientation of SAR Imagery

The RD model is usually considered as the rigorous sensor model for SAR images, which is composed of three equations: the Range equation, the Doppler equation and the Earth model equation as follows:

$$\left\{\begin{array}{c}R=|R_s-R_T|\hfill \\ f_{Dc}=\frac{2}{\lambda |R_s-R_T|}(V_s-V_T)(R_s-R_T)\hfill \\ \frac{X^2+Y^2}{{(R_e+h)}^2}+\frac{Z^2}{R_p^2+h}=1,\hfill \end{array}\right.$$

(6)

where R represents the measurement range between the target and the sensor, $R_T$ and $R_S$ are position vectors of the target and SAR sensor, respectively; $V_T$ and $V_S$ serve as corresponding velocity vectors of the target and SAR sensor, $f_{Dc}$ denotes the Doppler centroid frequency; $(X,Y,Z)$ denotes the target position; h is the target height relative to the surface of the earth, $R_e$ and $R_p$ are the equatorial radius and the polar radius of the Earth [32].

According to Equation (6), the geolocation accuracy of SAR images are mainly influenced by the slant range measurement error, azimuthal time error, Doppler center frequency error, ephemeris error of the satellite platform, and the topographic error [30]. Therefore, the calibration of SAR images can be complicated due to different error sources.The slant range measurement error and azimuthal time error play an important role among all of these factors, which leads to geolocation errors in the X and Y direction, respectively. Especially, the slant range measurement error is associated with different combinations of bandwidth and pulse width of the SAR sensor, which can be calibrated with a simple static model by correcting the internal calibration time delay error and atmospheric time delay error [39].

After the calibration of slant range measurement error, the geolocation accuracy of SAR images can be improved by several meters.The coarse-calibration can be achieved by updating sensor parameters of the RD model. Normally, the geolocation accuracy of coarse-calibrated SAR imagery cannot meet the requirement for further photogrammetric applications. Traditional methods for geolocation accuracy improvement of SAR images usually depend on the assistance of additional reference data—such as GCPs and LiDAR data. The collection of these reference data usually requires considerable financial and human resources. Hence, a free combined adjustment model is proposed for the production of remote sensing data with very-high geolocation accuracy.

3.3. Combined Adjustment Model

3.3.1. Unification of Coordinate System

Previous studies have revealed different types of combined adjustment models for multiple sources remote sensing data. Most of them are developed based on the RFM, and almost all methods highly depend on some additional existed reference data as mentioned above. Differently, we proposed a new combined adjustment model designed for the geometric calibration of SAR and optical images.

For optical images, interior and exterior orientation parameters are necessary for the establishment of the Collinear Condition Equations [44]. Considering the complexity and inconsistency between different platforms, coefficients of the RFM are provided by image vendors as a substitution. Usually, the production of RPCs is conducted under the Geodetic Coordinate System, whereas the RD model is defined based on the Cartesian Coordinate System. Therefore, parameters of the RD model are transformed into the Geodetic Coordinate System according to Equation (7).

$$\begin{array}{cc}\hfill X& =(N+H)cos\varphi cos\lambda \hfill \\ \hfill Y& =(N+H)cos\varphi sin\lambda \hfill \\ \hfill Z& =[N(1-e^2)+H]sin\varphi ,\hfill \end{array}$$

(7)

where $(X,Y,Z)$ denote the space rectangular coordinates; $\varphi$ and $\lambda$ are the radians of the geodetic latitude and longitude coordinates; N represents the Earth Curve Radius with

$$\begin{array}{c}\hfill N=\frac{R_e}{\sqrt{1-e^2{sin}^2\varphi }},\end{array}$$

(8)

where e denotes the Earth Curve.

Giving the relationship between the geodetic coordinates and space rectangular coordinates, parameters of the RD model can be translated into the Geodetic Coordinate System. Differently, the RD model are developed based on the Range Equation and Doppler Equation, which indicates that the geolocation accuracy should be reprojected into the image space to keep in line with the RFM.

3.3.2. Combined Normal Equations

As demonstrated in Equation (3), an affine transformation model can compensate for the geolocation error of the RFM efficiently. The situation can be much more complicated when it comes to the RD model because we cannot develop a formula that indicates the relationship between the object-space coordinates and image-space coordinates directly. As demonstrated above, the slant range measurement error and azimuthal time delay error are the main factors that influence the reprojection error of the image-space coordinates. For simplicity, the RD model can be rewritten as follows:

$$\begin{array}{cc}\hfill G_r& =(V_s-V_T)(R_s-R_T)-\frac{\lambda f_{Dc}}{2}·R\hfill \\ \hfill G_c& =|R_s-R_T|-R,\hfill \end{array}$$

(9)

where $G_r$ and $G_c$ are influenced by the image-space coordinates. To simplify the normal equations, a traditional “stop and go” assumption is utilized here [45]. Hence, parameters of the RD model can be represented by the image-space coordinates as:

$$\begin{array}{cc}\hfill R& =R_{near}+[(x+dx)·c_s]/(2·f_s)\hfill \\ \hfill R_{ref}& =R_{near}+W/2·w_s\hfill \\ \hfill d{t}_0& =(R-R_{ref})/c_s,d{t}_1=t_0+(y+dy)/prf\hfill \\ \hfill f_{DC}& =\sum _{i=0}^l{d}_i·d{t}_0^i\hfill \\ \hfill R_S& =\sum _{i=0}^n{p}_i·d{t}_1^i,V_S=\sum _{i=0}^m{q}_i·d{t}_1^i,\hfill \end{array}$$

(10)

where $(x,y)$ represent the image-space coordinates; $(dx,dy)$ are the corresponding corrections; $R_{near}$ is the measured range corresponding to the first pixel in the image sample direction; $t_0$ denotes the start imaging time in the image line direction; $c_s$ represents the speed of light; $f_s$ is the sampling frequency in the slant range direction; $R_{ref}$ denotes the reference slant range distance, $d{t}_0$ is the reference of the Doppler time; $d_i$ represent the coefficients of the Doppler center frequency; W and $w_s$ are the image width and widthspace; $d{t}_1$ denotes the azimuth time; $prf$ is the pulse repetition frequency; $p_i$ and $q_i$ are coefficients to fitting the position and velocity vector of the satellite platform.

Hence, normal equations assembling the RD model and RFM can be developed as:

$$\begin{array}{cc}\hfill F_r& =a_0+a_1·s+a_2·l-r\hfill \\ \hfill F_c& =b_0+b_1·s+b_2·l-c\hfill \\ \hfill G_r& =(V_s-V_T)·(R_s-R_T)-\frac{\lambda f_{Dc}}{2}·R\hfill \\ \hfill G_c& =|R_s-R_T|-R.\hfill \end{array}$$

(11)

Assuming the number of optical and SAR images are m and n, the coefficient matrix of normal equations can be obtained in the form of Equations (12) and (13).

$$A=\left[\begin{array}{ccccccccc}\frac{\partial F_r^1}{\partial a_0^1}& 0& 0& 0& \cdots & 0& 0& 0& 0\\ 0& \frac{\partial F_c^1}{\partial b_0^1}& 0& 0& \cdots & 0& 0& 0& 0\\ 0& 0& \frac{\partial G_r^1}{\partial y_1}& \frac{\partial G_r^1}{\partial x_1}& \cdots & 0& 0& 0& 0\\ 0& 0& \frac{\partial G_c^1}{\partial y_1}& \frac{\partial G_c^1}{\partial x_1}& \cdots & 0& 0& 0& 0\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮& ⋮& ⋮\\ 0& 0& 0& 0& \cdots & \frac{\partial F_r^n}{\partial a_0^n}& 0& 0& 0\\ 0& 0& 0& 0& \cdots & 0& \frac{\partial F_c^n}{\partial b_0^n}& 0& 0\\ 0& 0& 0& 0& \cdots & 0& 0& \frac{\partial G_r^m}{\partial y_m}& \frac{\partial G_r^m}{\partial x_m}\\ 0& 0& 0& 0& \cdots & 0& 0& \frac{\partial G_c^m}{\partial y_m}& \frac{\partial G_c^m}{\partial x_m}.\end{array}\right]$$

(12)

$$B=\left[\begin{array}{cccccccccc}\frac{\partial F_r^1}{\partial P}& \frac{\partial F_r^1}{\partial L}& \frac{\partial F_r^1}{\partial H}& 0& 0& 0& \cdots & 0& 0& 0\\ \frac{\partial F_c^1}{\partial P}& \frac{\partial F_c^1}{\partial L}& \frac{\partial F_c^1}{\partial H}& 0& 0& 0& \cdots & 0& 0& 0\\ \frac{\partial G_r^1}{\partial P}& \frac{\partial G_r^1}{\partial L}& \frac{\partial G_r^1}{\partial H}& 0& 0& 0& \cdots & 0& 0& 0\\ \frac{\partial G_c^1}{\partial P}& \frac{\partial G_c^1}{\partial L}& \frac{\partial G_c^1}{\partial H}& 0& 0& 0& \cdots & 0& 0& 0\\ 0& 0& 0& \frac{\partial F_r^2}{\partial P}& \frac{\partial F_r^2}{\partial L}& \frac{\partial F_r^2}{\partial H}& \cdots & 0& 0& 0\\ 0& 0& 0& \frac{\partial F_r^2}{\partial P}& \frac{\partial F_r^2}{\partial L}& \frac{\partial F_r^2}{\partial H}& \cdots & 0& 0& 0\\ 0& 0& 0& \frac{\partial G_r^2}{\partial P}& \frac{\partial G_r^2}{\partial L}& \frac{\partial G_r^2}{\partial H}& \cdots & 0& 0& 0\\ 0& 0& 0& \frac{\partial G_r^2}{\partial P}& \frac{\partial G_r^2}{\partial L}& \frac{\partial G_r^2}{\partial H}& \cdots & 0& 0& 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮& ⋮\\ 0& 0& 0& 0& 0& 0& \cdots & \frac{\partial F_r^n}{\partial P}& \frac{\partial F_r^n}{\partial L}& \frac{\partial F_r^n}{\partial H}\\ 0& 0& 0& 0& 0& 0& \cdots & \frac{\partial F_r^n}{\partial P}& \frac{\partial F_r^n}{\partial L}& \frac{\partial F_r^n}{\partial H}\\ 0& 0& 0& 0& 0& 0& \cdots & \frac{\partial G_r^n}{\partial P}& \frac{\partial G_r^n}{\partial L}& \frac{\partial G_r^n}{\partial H}\\ 0& 0& 0& 0& 0& 0& \cdots & \frac{\partial G_r^n}{\partial P}& \frac{\partial G_r^n}{\partial L}& \frac{\partial G_r^n}{\partial H}.\end{array}\right]$$

(13)

Partial derivatives of optical images based on the RFM can be easily obtained, whereas the formula corresponding to $G_r$ and $G_c$ are more complicated. With the help of Equation (11), partial derivatives including $\frac{\partial G_r}{\partial x}$, $\frac{\partial G_r}{\partial y}$, $\frac{\partial G_c}{\partial x}$ and $\frac{\partial G_c}{\partial y}$ can be derived as:

$$\begin{array}{cc}\hfill \frac{\partial G_r}{\partial x}& =-\frac{\lambda f_{Dc}}{2}·\frac{\partial R}{\partial x}-\frac{\lambda R}{2}·\frac{\partial f_{DC}}{\partial d{t}_0}·\frac{\partial d{t}_0}{\partial R}·\frac{\partial R}{\partial x}=-\frac{\lambda f_{Dc}}{2}·\frac{c_s}{2f_s}-\frac{\lambda R}{2}·\frac{1}{2f_s}·\sum _{i=0}^l{d}_i·d{t}_0^{i-1}\hfill \\ \hfill \frac{\partial G_r}{\partial y}& =(V_S-V_T)·\frac{\partial R_S}{\partial d{t}_1}·\frac{\partial d{t}_1}{\partial y}+(R_S-R_T)·\frac{\partial V_S}{\partial d{t}_1}·\frac{\partial d{t}_1}{\partial y}\hfill \\ \hfill & =(V_S-V_T)·\frac{1}{prf}·\sum _{i=1}^n{p}_i·d{t}_1^{i-1}+(R_S-R_T)·\frac{1}{prf}·\sum _{i=1}^m{q}_i·d{t}_1^{i-1}\hfill \\ \hfill \frac{\partial G_c}{\partial x}& =\frac{\partial R}{\partial x}=\frac{c_s}{2fs}\hfill \\ \hfill \frac{\partial G_c}{\partial y}& =\frac{\partial R_S}{\partial d{t}_1}·\frac{d{t}_1}{y}=\frac{1}{prf}·\sum _{i=1}^n{p}_i·d{t}_1^{i-1}.\hfill \end{array}$$

(14)

The partial derivatives from $G_r$ and $G_c$ to P, L and H can be represented as Equation (15).

$$\begin{array}{ccc}\hfill \frac{\partial G_r}{\partial P}& =& \frac{\partial G_r}{\partial X}\frac{\partial X}{\partial \varphi }\frac{\partial \varphi }{\partial P}+\frac{\partial G_r}{\partial Y}\frac{\partial Y}{\partial \varphi }\frac{\partial \varphi }{\partial P}+\frac{\partial G_r}{\partial Z}\frac{\partial Z}{\partial \varphi }\frac{\partial \varphi }{\partial P}\hfill \\ & =& -\frac{\pi }{180}·[(V_S.x-V_T.x)\frac{\partial X}{\partial \varphi }+(V_S.y-V_T.y)\frac{\partial Y}{\partial \varphi }+(V_S.z-V_T.z)\frac{\partial Z}{\partial \varphi }]\hfill \\ \hfill \frac{\partial G_r}{\partial L}& =& \frac{\partial G_r}{\partial X}\frac{\partial X}{\partial \lambda }\frac{\partial \lambda }{\partial L}+\frac{\partial G_r}{\partial Y}\frac{\partial Y}{\partial \lambda }\frac{\partial \lambda }{\partial L}+\frac{\partial G_r}{\partial Z}\frac{\partial Z}{\partial \lambda }\frac{\partial \lambda }{\partial L}\hfill \\ & =& -\frac{\pi }{180}·[(V_S.x-V_T.x)\frac{\partial X}{\partial \lambda }+(V_S.y-V_T.y)\frac{\partial Y}{\partial \lambda }+(V_S.z-V_T.z)\frac{\partial Z}{\partial \lambda }]\hfill \\ \hfill \frac{\partial G_r}{\partial H}& =& \frac{\partial G_r}{\partial X}\frac{\partial X}{\partial H}+\frac{\partial G_r}{\partial Y}\frac{\partial Y}{\partial H}+\frac{\partial G_r}{\partial Z}\frac{\partial Z}{\partial H}\hfill \\ & =& (V_S.x-V_T.x)\frac{\partial X}{\partial H}+(V_S.y-V_T.y)\frac{\partial Y}{\partial H}+(V_S.z-V_T.z)\frac{\partial Z}{\partial H}\hfill \\ \hfill \frac{\partial G_c}{\partial P}& =& \frac{\partial G_c}{\partial X}\frac{\partial X}{\partial \varphi }\frac{\partial \varphi }{\partial P}+\frac{\partial G_c}{\partial Y}\frac{\partial Y}{\partial \varphi }\frac{\partial \varphi }{\partial P}+\frac{\partial G_c}{\partial Z}\frac{\partial Z}{\partial \varphi }\frac{\partial \varphi }{\partial P}\hfill \\ & =& \frac{\pi }{180·|R_s-R_T|}·[(R_S.x-R_T.x)\frac{\partial X}{\partial \varphi }+(R_S.y-R_T.y)\frac{\partial Y}{\partial \varphi }+(R_S.z-R_T.z)\frac{\partial Z}{\partial \varphi }]\hfill \\ \hfill \frac{\partial G_c}{\partial L}& =& \frac{\partial G_c}{\partial X}\frac{\partial X}{\partial \lambda }\frac{\partial \lambda }{\partial L}+\frac{\partial G_c}{\partial Y}\frac{\partial Y}{\partial \lambda }\frac{\partial \lambda }{\partial L}+\frac{\partial G_c}{\partial Z}\frac{\partial Z}{\partial \lambda }\frac{\partial \lambda }{\partial L}\hfill \\ & =& \frac{\pi }{180·|R_s-R_T|}·[(R_S.y-R_T.y)\frac{\partial X}{\partial \lambda }+(R_S.y-R_T.y)\frac{\partial Y}{\partial \lambda }+(R_S.z-R_T.z)\frac{\partial Z}{\partial \lambda }]\hfill \\ \hfill \frac{\partial G_c}{\partial H}& =& \frac{\partial G_c}{\partial X}\frac{\partial X}{\partial H}+\frac{\partial G_c}{\partial Y}\frac{\partial Y}{\partial H}+\frac{\partial G_c}{\partial Z}\frac{\partial Z}{\partial H}\hfill \\ & =& \frac{1}{|R_s-R_T|}·[(R_S.x-R_T.x)\frac{\partial X}{\partial H}+(R_S.y-R_T.y)\frac{\partial Y}{\partial H}+(R_S.z-R_T.z)\frac{\partial Z}{\partial H}].\hfill \end{array}$$

(15)

Given Equation (8), partial derivatives of X, Y and Z to $\varphi$, $\lambda$ and H are easy to be derived. Hence, normal equations can be established in the form of matrices as:

$$\begin{array}{cc}\hfill V_O& =A_O·X_{A_O}+B_O·X_{B_O}-l_O,P_O\hfill \\ \hfill V_S& =A_S·X_{A_S}+B_S·X_{B_S}-l_S,P_S,\hfill \end{array}$$

(16)

where the subscript O and S represent matrices designed for optical and SAR images, respectively.

The establishment of the combined normal equations provides a generic way for the geometric processing of SAR and optical imagery. However, the absolute geolocation accuracy after free block adjustment cannot meet our requirements without the help of GCPs. Therefore, the heterogeneous weight strategy, defined as $P_O$ and $P_S$, is introduced for better convergence.

3.3.3. Heterogeneous Weight Strategy

Traditional block adjustment methods are developed with an identify weight matrix, which indicates that the contribution of all elements involved is the same. However, the geometric performance of multiple sources remote sensing data varies greatly according to different platforms. Generally, the performance of most world-class SAR imagery can achieve better than 10 m, whereas the geolocation accuracy of different optical imagery ranging from several meters to hundreds of meters. Therefore, the heterogeneous weight strategy is proposed to ensure that images with higher accuracy will contribute more during the combined adjustment process, which ensures an optimum result will be obtained without using GCPs.

Different from traditional identify weight matrix, the heterogeneous weight matrix composed of $P_O$ and $P_S$ are defined as follows:

$$P=\left[\begin{array}{ccccccccc}{P}_{O_r}^0& 0& 0& 0& \cdots & 0& 0& 0& 0\\ 0& P_{O_c}^0& 0& 0& \cdots & 0& 0& 0& 0\\ 0& 0& P_{S_r}^0& 0& \cdots & 0& 0& 0& 0\\ 0& 0& 0& P_{S_c}^0& \cdots & 0& 0& 0& 0\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮& ⋮& ⋮\\ 0& 0& 0& 0& \cdots & P_{O_r}^n& 0& 0& 0\\ 0& 0& 0& 0& \cdots & 0& P_{O_c}^n& 0& 0\\ 0& 0& 0& 0& \cdots & 0& 0& P_{S_r}^m& 0\\ 0& 0& 0& 0& \cdots & 0& 0& 0& P_{S_c}^m\end{array}\right],\left\{\begin{array}{cc}\hfill P_{O_r}& =m_O·\frac{1}{q}·\frac{1}{H·tan\theta }·\frac{1}{A_O}\hfill \\ \hfill P_{O_c}& =m_O·\frac{l^o}{l_{sum}^o}·\frac{1}{q}·\frac{1}{H·tan\omega }·\frac{1}{C_O}\hfill \\ \hfill P_{S_r}& =m_S·\frac{1}{q}·\frac{1}{A_S}\hfill \\ \hfill P_{S_c}& =m_S·\frac{l^s}{l_{sum}^s}·\frac{1}{q}·\frac{tan{\theta }^{\prime }}{C_S},\hfill \end{array}\right.$$

(17)

where the subscripts O and S represent designed for optical and SAR imagery, respectively; m denotes the adaptive parameter to keep balance between different weights; q is the resolution of each involved image; H is the height of the optical satellite, $\theta$ and $\omega$ are measured rolling angle and pitching angle of the optical imaging sensor; ${\theta }^{\prime }$ represent the looking angle of the SAR sensor; l denotes the number of images divided into groups according to different principles; $l_{sum}$ is the total number of involved optical/SAR images; C and A represent the relative geolocation error of each image computed during each iteration.

Without the help of GCPs, the above strategies provide a generic guidance for the determination of an optimum weight for each observation. In practice, the determination of $m_O$ and $m_S$ is conducted based on more than one test. A converged solution will be obtained with the aid of some popular modified Least-Square method, and further photogrammetric applications can be investigated after the fine-calibration of the multiple sources dataset.

4. Experimental Results and Analysis

4.1. Experimental Dataset

Considering the revisited period of different commercial satellites with very high resolution, the collection of multiple sources and multiple observation dataset is time-consuming and expensive. In comparison, the comparison of some open access datasets, such as the Sentinel-1 data, are much lower. Hence, multiple remote sensing images obtained from the Jilin-1 (JL-1) optical small satellite constellation and the Gaofen-3 (GF-3) SAR satellite are involved to verify the efficiency of our proposed method. As the first commercial optical satellite constellation, it is composed of 14 small satellites by the end of 2020. The resolution of JL-1 optical images is 0.92 m and the swath width is 11 km. Benefited from the non-fixed camera on the platform, images can be obtained at different imaging times and looking angle, which provides multiple observation dataset with more information. The GF-3 satellite is the first civilian microwave remote sensing imaging satellite. The nominal resolution of obtained GF-3 images varies from 1 m to 500 m with the swath width ranging from 10 km to 650 km. In this experiment, 7 JL-1 optical images obtained from 4 different platforms and 3 GF-3 SAR images covering a rural area around the Mount Song area are selected. Detailed information is listed in

Table 1. Detailed information of experimental dataset (DEC means descending, and ASC is the abbreviation of ascending).

Platform	Acquisition Date	Orbit	Incidence Angle (${}^{\circ }$)	Size (Pixels)	Resolution (m)
JL-104-1	2 April 2018	DEC	8.73	11,516 × 12,143	0.93 × 0.94
JL-104-2	23 June 2018	DEC	4.33	11,506 × 12,148	0.92 × 0.92
JL-104-3	29 October 2018	DEC	0.84	11,518 × 12,120	0.92 × 0.92
JL-105-1	15 June 2018	DEC	−0.52	11,518 × 12,056	0.92 × 0.92
JL-105-2	10 October 2018	DEC	5.19	11,518 × 12,008	0.92 × 0.94
JL-106-1	7 June 2018	DEC	1.38	11,530 × 12,007	0.92 × 0.92
JL-107-1	31 March 2018	DEC	1.97	11,513 × 11,991	0.92 × 0.92
GF3-1	28 December 2016	DEC	37.43	16,215 × 21,531	2.24 × 2.86
GF3-2	14 November 2019	DEC	24.68	21,625 × 23,354	1.12 × 2.61
GF3-3	20 December 2019	ASC	41.43	18,124 × 20,316	2.24 × 3.03

, and the geometric distribution can be found in Figure 2.

Influenced by the imaging modality, targets in SAR images are difficult to be identified compared with optical images. Therefore, 5 check point sets are extracted from an existed database of control points. All check points located in the corner of border areas or road intersections. Moreover, 147 tie points are extracted automatically based on an efficient multiple source image matching method [43]. Figure 3 shows the geographical distribution of involved optical and SAR images, as well as the distribution of extracted corresponding tie points.

4.2. Performance of the Combined Adjustment

Before the adjustment process, the slant range measurement error is firstly calibrated based on our previous statistic results [39].

Table 2. Geometric performance of GF-3 Synthetic Aperture Radar (SAR) images before and after coarse calibration (pixels).

Images	Before Calibration		After Calibration
	X	Y	X	Y
GF3-1	11.11	8.09	1.56	8.09
GF3-2	10.96	7.84	1.73	7.84
GF3-3	12.37	8.21	1.61	8.21

gives the geolocation results before and after the coarse-calibration. The initial geolocation accuracy of both GF-3 SAR images is approximately 11 pixels in the X direction and 8 pixels in the Y direction, which is in accordance with previous studies [25]. After calibration, the geolocation error in the X direction influenced by the slant range measurement error is eliminated greatly, whereas the variation of geolocation error in the Y direction is negligible. Based on the coarse-calibration step, the geolocation accuracy of both GF-3 SAR images is improved significantly, which guarantee a better result of the whole combined adjustment process.

To verify the efficiency of our proposed combined adjustment model, a traditional RFM based adjustment process is conducted in comparison. Hence, the RPCs of SAR images need to be produced in advance. Generally, a terrain-dependent method relying on well-distributed GCPs performs the best [46]. In contrast, the terrain-independent method is commonly applied with the help of an open-source DEM. Based on the established spatial grid, the fitting accuracy of produced RPCs can reach sub-pixels [47]. Hence, RPCs of GF-3 SAR images are produced after coarse-calibration and the RFM based combined adjustment model including all SAR and optical images can be developed according to Equation (4).

Figure 4 gives the relative geolocation accuracy between optical and SAR images after combined adjustment. Without the help of GCPs, geolocation results after free combined adjustment cannot meet the requirement for further processing, such as the production of geocoded 3D products and target localization. Hence, the heterogeneous weight strategy is applied for better convergence. The final results are listed in

Table 3. Geometric performance of the whole dataset with different combined adjustment methods (pixels).

Items	Before Adjustment		After Adjustment		After Weighted Adjustment
	X	Y	X	Y	X	Y
RFM	146.36	111.75	61.97	72.23	2.97	4.78
Proposed	146.36	111.75	59.43	71.89	2.65	4.43

. After processing, the geolocation accuracy increases to approximately 3 pixels in the X direction and 4.5 pixels in the Y direction. At the same time, our proposed model also shows better performance than the traditional RFM based combined adjustment model. Furthermore, Figure 5 shows the error distribution after processed by these two methods. Compared with the traditional one, our proposed model also gives the best performance in convergence, which can be derived from the consistency between images.

Table 3 gives the root mean square error (RMSE) of the whole dataset processed with different models. The RFM based combined adjustment model improves the geolocation accuracy of all datasets from about 146.36 pixels in the X direction and 111.75 pixels in the Y direction to 61.97 pixels in the X direction and 72.23 pixels in the Y direction with an identity weight matrix. In contrast, the performance of our proposed model is better than the traditional one, with an accuracy of 59.43 pixels in the X direction and 71.89 pixels in the Y direction.

5. Discussions and Conclusions

Traditional methods for the geometric processing of multiple sources optical and SAR imagery are developed based on the RFM. Different from most optical satellite imagery, RPCs of SAR images are not always provided by image vendors, especially for some popular SAR sensors such as the TerraSAR-X and Sentinel-1 satellite. Therefore, the production of RPCs for SAR imagery has to be produced by users additionally. Moreover, the fitting accuracy is highly dependent on the terrain.

Aiming at finding a generic and simple way for geometric calibration of multiple sources optical and SAR images, we proposed a new combined adjustment model. Unlike traditional RFM-based methods, the slant range measurement error of SAR images obtained from the GF-3 satellite is calibrated based on our previous work. After the coarse-calibration step, tie points are automatically extracted from both optical and SAR images. The combined adjustment model is established by reprojecting parameters of the RD model into the same coordinate system with the RPCs. Together with an additional heterogeneous weight strategy, our proposed model gives the best performance. Compared with traditional methods, our proposed model provides a new way for the integration of multiple sources optical and SAR data, which do not introduce extra fitting accuracy. Further, this proposed model also enables the application for precise photogrammetric reconstruction.