2OC: A General Automated Orientation and Orthorectification Method for Corona KH-4B Panoramic Imagery

Abstract

Due to a lack of geographical reference information, complex panoramic camera models, and intricate distortions, including radiation, geometric, and land cover changes, it can be challenging to effectively apply the large number (800,000+) of high-resolution Corona KH-4B panoramic images from the 1960s and 1970s for surveying-related tasks. This limitation hampers their significant potential in the remote sensing of the environment, urban planning, and other applications. This study proposes a method called 2OC for the automatic and accurate orientation and orthorectification of Corona KH-4B images, which is based on generalized control information from reference images such as Google Earth orthophoto. (1) For the Corona KH-4B panoramic camera, we propose an adaptive focal length variation model that ensures accuracy and consistency. (2) We introduce a robust multi-source remote sensing image matching algorithm, which includes an accurate primary orientation estimation method, a multi-threshold matching enhancement strategy based on scale, orientation, and texture (MTE), and a model-guided matching strategy. These techniques are employed to extract high-accuracy generalized control information for Corona images with significant geometric distortions and numerous weak texture areas. (3) A time-iterative Corona panoramic digital differential correction method is proposed. The orientation and orthorectification results of KH-4B images from multiple regions, including the United States, Russia, Austria, Burkina Faso, Beijing, Chongqing, Gansu, and the Qinghai–Tibet Plateau in China, demonstrate that 2OC not only achieves automation but also attains a state-of-the-art level of generality and accuracy. Specifically, the standard deviation of the orientation is less than 2 pixels, the mosaic error of orthorectified images is approximately 1 pixel, and the standard deviation of ground checkpoints is better than 4 m. In addition, 2OC can provide a longer time series analysis of data from 1962 to 1972, benefiting various fields such as environmental remote sensing and archaeology.

1. Introduction

The Corona program, as a reconnaissance satellite initiative aimed at acquiring military strategic and weapon intelligence, collected over 860,000 images of the Earth’s surface [1,2]. The highest-quality images from the Corona program, the KH-4B images [3], have a resolution of 1.8 m, which can be used for the identification and mapping of historic road systems [4], architectural structures [5], and historic landscapes [6]. They are also valuable for studying urban expansion [7], creating historical land cover maps [8,9], reconstructing historical ecological data [10], and assessing glacier area changes [11]. Nevertheless, the original KH-4B images suffer from intricate panoramic geometric distortions due to the absence of georeferencing, preventing their direct use in scientific research. Moreover, because of complex distortions, such as radiation distortion, geometric distortion, land cover changes, and weak texture, the current georeferencing process for KH-4B imagery is achieved through expensive and time-consuming manual rectification, significantly limiting their practical utility.

In recent years, some scholars [12] have attempted to fit the Corona panoramic camera model using traditional frame-based camera models to correct the panoramic distortions. However, the rectification accuracy of these methods is low because of significant differences in camera models. To address this issue, Jacobsen et al. [13] proposed a perspective framework camera model that incorporates geometric panoramic transformation terms. This model was applied on a large scale and achieved an orientation mean error of approximately 11.4 pixels. A fisheye camera model was used by Lauer et al. [14] to fit the panoramic distortions of Corona images, achieving a planimetric positioning accuracy of 17 m on local Corona images. Additionally, Sohn [15] proposed the use of a second-order rational function model (RFM) to georectify Corona KH-4B images. However, the limited number of manually extracted generalized control points resulted in lower accuracy. Sohn [15] also proposed two rigorous mathematical models, which are a modified projection model based on the frame-based model and a time-related exterior orientation elements projection model. The time-related exterior orientation elements projection model, as a complex and rigorous panoramic camera model, achieved sub-2-pixel orientation accuracy on a local region with around 30 manually extracted generalized control points. However, this camera model cannot adapt to focal length variations, and the experiments show that the official design focal length is not always optimal for all image orientations, leading to increased errors and even orientation failure. Furthermore, the aforementioned approaches had a significant drawback: the need for manual extraction of generalized control points [16]. We aim to automatically extract match points between KH-4B images and reference images using image matching techniques. However, traditional image matching techniques (such as SIFT [17], SURF [18], and UR-SIFT [19]) are unable to extract stable and accurate control information from Corona images due to the complexity of multiple distortions arising from differences in satellite orbits, sensor characteristics, and large temporal gaps. For instance, Bhattacharya et al. [20] manually selected generalized control points and used remote sensing software (Graz) to generate a Corona KH-4B digital elevation model (DEM). These methods improve orientation accuracy with enhanced mathematical models but are only suitable for small datasets because of their reliance on manually extracted control information.

This study proposes a universal method, named 2OC, for the automatic orientation and orthorectification of Corona KH-4B images. First, a 14-parameter panoramic mathematical model and a time-iterative orthorectification technique are introduced to correct the panoramic geometric distortion and fit the focal length distortions of Corona panoramic images. Second, to address the complex distortions of Corona images, a robust image-matching method is proposed to extract correspondences between Corona images and reference images, generating generalized control points. Specifically, (1) A robust feature matching algorithm, NIFT, is proposed to estimate the transformation relationship between images, considering radiation and geometry distortions. (2) A multi-threshold matching enhancement strategy, MTE, is developed to optimize the distribution and quantity of generalized control points in areas with weak textures and land cover changes, thus improving the overall accuracy of the control information. (3) A model-guided matching strategy is introduced to reduce the impact of panoramic distortions.

The main contributions of this study are as follows:

• We propose an automatic orientation and orthorectification method (2OC) for Corona KH-4B images. To validate its effectiveness, we apply 2OC to a large number of multi-regional KH-4B images with various sources of references, and the detailed information is given in Table 1. The orientation accuracy is better than 2 pixels, the stitching error of orthorectified images is approximately 1 pixel, and the ground checkpoints have an RMSE accuracy better than 4 m. This demonstrates that 2OC is capable of processing KH-4B images with different regions, terrains, and image distortions using multiple reference images.
• We propose a 14-parameter corona panoramic camera model and a time-iterative panoramic orthorectification method. First, to address the focal length and panoramic distortions of KH-4B images, we propose a 14-parameter panoramic camera model. Second, to obtain an analytical solution for back-projecting ground coordinates to image coordinates, we propose a novel time-iterative panoramic orthorectification method.
• We introduce a robust control information extraction algorithm for extracting match points between KH-4B images and reference images to generate control information. To overcome land cover changes and geometric distortions (rotation, scale, panoramic), we propose a robust feature matching algorithm, a multi-threshold matching enhancement strategy (MTE) based on local texture, scale, and orientation, and a model-guided matching strategy.

Table 1. The details of the KH-4B images and the reference images used in the experiments.

KH-4B Scenes		Reference Image	Distortion
Region	Terrain
USA, Russia, Ethiopia, Burkina Faso, and China’s Beijing, Chongqing, Gansu, and the Qinghai–Tibet plateau	Loess plateau, glacier, plain, hill, high mountain	Google Earth, Bing, and ArcGIS	Temporal disparities (50-year), scale variations (1:5), rotational disparities (0–360 degrees), radiometric differences, and local land cover alterations

2. Related Work

In this section, we first provide an overview of current methods for processing Corona images and then briefly review the matching techniques for multi-source remote sensing images.

2.1. Geometric Processing of the Corona Images

Non-panoramic camera model-based methods: Altmaier et al. [12] directly used frame-based image processing software (ERDAS IMAGE OrthoBASE Pro) to process Corona images, obtaining digital surface models with elevation and planimetric accuracy of 10 m and 3 m, respectively. Casana [21] also used this method to process Corona images in the Middle East and achieved orientation errors of approximately 5 pixels within a small image size of less than 5000 × 5000 pixels. Nita et al. [22] utilized match points between panoramic images for relative orientation, followed by absolute orientation using manually extracted generalized control points to generate DSMs and orthorectified images with a planimetric error of around 14 m. Rizayeva et al. [8] applied the method presented in [22] to produce orthorectified images with a resolution of 2.5 m and achieved a planimetric error of 16.3 ± 10.4 m. Furthermore, Bhattacharya et al. [20] used Graz (RSG) software to process KH-4B images, resulting in a triangulation error of approximately 2.5 pixels. Moreover, Jacobsen et al. proposed a perspective framework camera model that incorporates panoramic transformation terms, but it exhibited a high standard deviation of 11.4 pixels in orientation. After all, non-panoramic camera models, although simple, suffer from lower accuracy.

Panoramic camera model-based methods: To better fit the Corona KH-4B panoramic camera, Sohn [15] proposed two approaches: (1) modifying the panoramic camera model based on the differences between frame-based imaging and panoramic imaging models by analyzing their transformation equations, (2) developing a time-dependent panoramic camera model by analyzing the panoramic imaging process and considering camera and platform motions. Based on the manually extracted generalized control points, they achieved orientation accuracy of approximately 1.5 pixels for small-scale KH-4B images. Additionally, Shin and Schenk [23] proposed a simplified panoramic camera model, assuming that the internal parameters of the camera only undergo motion along the sensor direction during exposure and the external parameters experience motion along the flight direction. They obtained a height error of approximately 12 m in Corona stereo pairs. Although the fisheye camera model is different from the Corona panoramic camera model, Lauer et al. [14] attempted to apply the fisheye camera model to process Corona images and achieved a planimetric accuracy of approximately 17 m. These methods improve orientation accuracy with enhanced mathematical models but are only suitable for small datasets because of their reliance on manually extracted control information.

2.2. Image Matching Techniques

In recent years, scale-invariant feature transform (SIFT) [17] has been widely used as a classical local feature extraction algorithm for image registration in remote sensing. However, the non-linear intensity distortion between multi-temporal and multi-sensor remote sensing images severely degrades the performance of SIFT. Therefore, Ye et al. proposed some region-based matching algorithms, such as HOPC [24] and CFOG [25], which have been successfully applied to multi-sensor image registration. These methods rely on prior information about the image position. To address this issue, Li et al. proposed feature-matching algorithms, namely RIFT [26] and LNIFT [27], which do not depend on prior location information. These algorithms exhibit good performance in combating non-linear radiometric distortion but have limited robustness regarding rotation and scale distortion. Additionally, with the rapid development of deep learning techniques, Ghuffar et al. [28] employed the deep model Superglue [29], designed for matching natural scene images, to automatically extract control information from Landsat images, achieving a sub-pixel level of median error. However, it cannot adapt well to the unique complex distortions in KH-4B images. Therefore, the current image-matching techniques are not robust enough to handle the complex distortions of KH-4B images for orientation and rectification tasks.

Based on this, 2OC is proposed for the orientation and orthorectification of Corona KH-4B images with complex distortions. First, a 14-parameter panoramic mathematical model and a digital differential orthorectification method are proposed to effectively fit the panoramic and focal length distortions of KH-4B images. Second, a robust image-matching algorithm is developed for the automatic extraction of control information. Extensive experimental results demonstrate that 2OC achieves orientation accuracy better than 2 pixels, with a mosaic error of approximately 1 pixel for orthorectified images and a median error of less than 4 m for ground checkpoints.

3. Corona KH-4B Image Processing

The 2OC process consists of multiple modules: a panoramic camera model, image orientation, orthorectification, and a generalized control information extraction algorithm. The detailed workflow is shown in Figure 1: First, the images are downsampled (❶), and a feature-matching algorithm is used to estimate the transformation matrix between KH-4B images and reference images (❷). Second, an image pyramid is constructed, and template matching (❸) and multi-threshold matching enhancement strategy (❸) are employed to obtain reliable match points (❹) for generating generalized control information (❹). Then, image orientation (❺) is performed, and model-guided matching (❻) is used to re-optimize the generalized control information (❻) and image orientation results (❼). Finally, the KH-4B orthorectified images are obtained using orthorectification based on iterative scanning time (❽).

Figure 1. The flowchart of 2OC. M represents the transformation matrix between images, and GCP is the abbreviation for generalized control points.

3.1. Introduction of the Corona Images

As shown in Table 2, the Corona missions include KH-1, KH-2, KH-3, KH-4, KH-4A, and KH-4B. According to U.S. Executive Order 12951 [30], these images were released to the National Archives and Records Administration (NARA) and the U.S. Geological Survey (USGS) on 23 February 1995. The complete panoramic image has a size of approximately 70 × 745 mm, and the USGS scanned the image at resolutions of 7 or 14 μm. However, due to the large size of the image, it was divided into four overlapping sections for scanning, labeled as a, b, c, and d, and generated four sub-images. Detailed information on Corona KH images is provided in Table 2.

Table 2. The detailed information of KH images.

Mission Designators	Film Size	Micron Size	File Size
KH-1; KH-2; KH-3	70 × 745 mm	12 Micron (1800 dpi)	80 MB (4 files)
KH-4; KH-4A; KH-4B		7 Micron (3600 dpi)	319 MB (4 files)

KH images have complex distortions, making them hard to process. (1) The Corona KH-4 camera rotates steadily in the across-track direction within an expansion of 70° while sequentially exposing a static film, obtaining a series of instantaneous strip images with significant panoramic distortions. (2) The KH-4B images may experience varying levels of deformation. Fortunately, the additional markings, panoramic geometry (PG) stripes, on the image can assist in evaluating the deformation. Specifically, during the photography process, the lamps mounted on the lens form straight lines at the edges of the image, as shown in Figure 2. Therefore, the PG stripes bend with film deformation. However, KH-4 and KH-4B images do not include PG stripes but rather feature shrinkage marks and format center indicators. (3) The scanning process of the image is not precisely calibrated, requiring the estimation of rotation and translation components using the overlapping areas of adjacent image blocks for accurate sub-image stitching. A previous study [28] has shown that there are varying levels of block-wise deformations within the images, which can result in incorrect sub-image transformations. These errors accumulate during the stitching process, affecting the overall accuracy. Therefore, we process the sub-images separately before stitching to avoid stitching errors and correct scanning errors, considering that the interior and exterior orientation elements of image orientation compensate for the rotation and translation of sub-images.

Figure 2. Schematic of film and PG reference data in KH-4B missions. All dimensions are in meters unless otherwise stated.

3.2. The Imaging Model of Corona KH-4B Panoramic Cameras

The imaging process of a KH-4B camera is depicted in Figure 3. While the satellite moves swiftly along its orbit, the camera rapidly rotates to sequentially expose the static film. This dynamic process results in time-varying exterior orientation elements of the camera. To better fit the imaging procedure of the KH-4B panoramic camera, a 14-parameter mathematical model is proposed. This model includes 12 exterior orientation elements associated with time, the dynamic correction parameter, and the image focal length. The derivation of the imaging model at any arbitrary time $t$ is provided below.

(1)

Exterior orientation of the KH-4B panoramic camera at time $t$

Figure 3. The imaging of KH-4B images. (a) The imaging process; (b) the imaging geometric relationship.

First, the change in exterior orientation elements, including position coordinate $\{X_{st}, Y_{st}, Z_{st}\}$ and orientation elements $\{{\omega }_t, {\phi }_t, {\kappa }_t\}$, caused by the satellite motion can be expressed using the following equations by assuming that they are linearly related to the time $t$.

$$X_{st}=X_{s0}+X_{s1}t$$

(1)

$$Y_{st}=Y_{s0}+Y_{s1}t$$

(2)

$$Z_{st}=Z_{s0}+Z_{s1}t$$

(3)

$${\omega }_t={\omega }_0+{\omega }_{1}t$$

(4)

$${\phi }_t={\phi }_0+{\phi }_{1}t$$

(5)

$${\kappa }_t={\kappa }_0+{\kappa }_{1}t$$

(6)

$$t=\frac{x_p}{L}$$

(7)

where $\left\{{\omega }_0,{\phi }_0,{\kappa }_0,X_{s0},Y_{s0},Z_{s0}\right\}$ and $\{{\omega }_t,{\phi }_t,{\kappa }_t,X_{st},Y_{st},Z_{st}\}$ are the exterior orientation elements at time 0 and $t$, where $tϵ\left[0, 1\right]$ after normalization. $\left\{{\omega }_1,{\phi }_1,{\kappa }_1,X_{s1},Y_{s1},Z_{s1}\right\}$ are the variation coefficients with respect to $t$. $x_p$ represents the coordinate of the instantaneous image on the panoramic image in the horizontal direction, $L$ represents the length of the film.

Second, we introduce a change in the exterior orientation elements caused by camera rotation along the cross-track direction with $t$, which is elaborated in Figure 3b and can be expressed using Equation (8).

$$\alpha =\frac{x_p}{f}$$

(8)

where $\alpha$ is the rotation angle in the cross-track direction, and $f$ is the camera focal length.

Therefore, the exterior orientation elements of the camera at $t$ are $\{X_{st},Y_{st},Z_{st},{\omega }_t,{\phi }_t+\alpha ,{\kappa }_t$}.

(2)

The imaging model of instantaneous strip images

Given that the instantaneous strip image width at time $t$ is extremely narrow, the instantaneous strip image satisfies the following collinearity equation:

$$\left[\begin{array}{c}0\\ y_p\\ -f\end{array}\right]=s{R}_{\alpha }{R}_t\left[\begin{array}{c}X-X_{0t}\\ Y-Y_{0t}\\ Z-Z_{0t}\end{array}\right]$$

(9)

$$R_{\alpha }=\left[\begin{array}{ccc}\cos \alpha & 0& \sin \alpha \\ 0& 1& 0\\ -\sin \alpha & 0& \cos \alpha \end{array}\right]$$

(10)

where $s$ represents the scale factor, $R_{\alpha }$ and $R_t$ are the rotation matrices caused by the camera rotation and satellite motion, respectively, where $R_t$ can be obtained using ${\omega }_t,{\phi }_t,{\kappa }_t$ in Equations (4)–(6). $\left(X,Y,Z\right)$ are the coordinates of a ground point, $y_p$ is the vertical coordinate of the corresponding image point.

Furthermore, the rapid motion along the track direction of the camera will produce dynamic deformation in the vertical direction during the exposure process. Considering this, we employ a displacement $y_{IMC}$ to mitigate this deformation, which can be expressed using Equation (11).

$$y_{IMC}=-Pf\sin \alpha \cos {\omega }_t$$

(11)

$$P=\frac{V}{H\delta }$$

(12)

where $V$ and $H$ represent the satellite’s velocity and orbit altitude, respectively. $\delta$ represents the angular velocity of the lens in the cross-track direction. Therefore, the imaging model can be described as follows.

$$\left[\begin{array}{c}0\\ y_p+y_{IMC}\\ -f\end{array}\right]=s{R}_{\alpha }{R}_t\left[\begin{array}{c}X-X_{0t}\\ Y-Y_{0t}\\ Z-Z_{0t}\end{array}\right]$$

(13)

By multiplying the left equation with $R_{\alpha }^T$

$$\left[\begin{array}{c}fsin\alpha \\ y_p+y_{IMC}\\ -fcos\alpha \end{array}\right]=s{R}_t\left[\begin{array}{c}X-X_{st}\\ Y-Y_{st}\\ Z-Z_{st}\end{array}\right]$$

(14)

If we define $N_x$, $N_y$ and $N_z$ as follows:

$$N_x=r_{11}\left(X-X_{st}\right)+r_{12}\left(Y-Y_{st}\right)+r_{13}\left(Z-Z_{st}\right)$$

(15)

$$N_y=r_{21}\left(X-X_{st}\right)+r_{32}\left(Y-Y_{st}\right)+r_{33}\left(Z-Z_{st}\right)$$

(16)

$$N_z=r_{31}\left(X-X_{st}\right)+r_{32}\left(Y-Y_{st}\right)+r_{33}\left(Z-Z_{st}\right)$$

(17)

where $r_{ij}$ represents the element at the i-th row and j-th column of the rotation matrix $R_t$. Moreover, the scale factor $s$ can be eliminated by dividing the first and second rows of the equation by the third row of the Equation (14).

$$\tan \alpha =-\frac{N_x}{N_y}$$

(18)

$$y_p+y_{IMC}=-f\cos \alpha \frac{N_y}{N_z}$$

(19)

Then, we can obtain the panoramic camera coordinates ($x_p,y_p$) based on the following collinearity function.

$$x_p=f{\tan }^{-1}\left(-\frac{N_x}{N_z}\right)$$

(20)

$$y_p=Pf\sin \alpha \cos {\omega }_{0t}-f\cos \alpha \frac{N_y}{N_z}$$

(21)

After all, the relationship between the ground coordinates ($X$, $Y$, and panoramic camera coordinates ($x_p,y_p$) can be modeled with the following 14 parameters: the camera’s initial exterior orientation parameters $\left\{X_{s0},Y_{s0},Z_{s0},{\omega }_0,{\phi }_0,{\kappa }_0\right\}$, coefficients for its linear variation over $t$ $\left\{X_{s1},Y_{s1},Z_{s1},{\omega }_1,{\phi }_1,{\kappa }_1\right\}$, image dynamic deformation coefficient $P$, and the camera focal length $f$.

3.3. Automated Orientation of Corona KH-4B Images Based on Generalized Control

We orient the KH-4B images by solving the 14 parameters using the generalized control points. Specifically, the image matching technique proposed in Section 4 is employed to automatically obtain a large number of well-distributed match points between KH-4B images and reference images. Here, we set the coordinates of a pair of match points as $\left(x_i^{\prime },y_i^{\prime }\right)$ in the KH-4B image and $\left(x_i^{″},y_i″\right)$ in the reference image. Using the geographic information of the reference image, the object coordinates $\left(X_i,Y_i\right)$ corresponding to $\left(x_i^{″},y_i^{\prime }\right)$ can be obtained, and the elevation information $Z_i$ can be obtained from the Digital Elevation Model (DEM). This results in the generalized control information $\left(x_i^{\prime },y_i\prime ,X_i,Y_i,Z_i\right)$.

Given that the imaging model has 14 parameters and each generalized control point can provide two equations, at least seven points are required to solve the parameters using Equations (20) and (21). When more generalized control points are available, these parameters can be solved using a least squares adjustment, which will enhance the calculation accuracy and reliability. Additionally, as the collinearity equations are non-linear, they must be linearized and require relatively accurate initial parameters.

However, Corona KH-4B images do not provide orientation parameters or auxiliary information that can be used to calculate the orientation parameters, such as the primary point coordinates, lens distortion coefficients, reference coordinates, satellite position, satellite velocity, and satellite attitude. As described in Section 3.1, we orient the sub-image separately instead of the whole image. Even though the primary point coordinates of the sub-image will deviate from the original photographic primary point; however, this deviation will be compensated by the external orientation elements of the pose parameters. In this study, the initial values of $X_{s0}$ and $Y_{s0}$ are set to the average geographical coordinates ($X_i$, $Y_i$) of all generalized control points, $Z_{s0}$ is set to 170,000 m based on the satellite orbit, w is set to {−15°, 15°} based on the forward and backward perspectives. Focal length f is set to 0.609602 m based on a default value provided by [28], and the other nine parameters are set to 0.

3.4. Orthorectification of KH-4B Panoramic Images Based on Iteration over Time $t$

Orthorectification is the process of mapping image points from panoramic KH-4B images to orthophoto. First, we calculate the corresponding ground coordinates $\left(X, Y, Z\right)$ for the image point $\left(x_o, y_o\right)$ of orthophoto. Then, we compute the corresponding panoramic coordinates $\left(x_p, y_p\right)$ for ground point $\left(X, Y, Z\right)$ based on the imaging model and the solved 14 parameters. However, the solution of $x_p$ require the exposure time t and the exterior orientation elements at $t$, which are unknown. To address the circular dependency problem, we formulate it as an optimization problem of minimizing an objective function $E$. $E$ describes the error between $\left(x_p, y_p\right)$ and the panoramic image coordinates $\left(x,y\right)$ computed based on ($X,Y,Z$), $t\left(x_p\right)$, and $\alpha \left(x_p\right)$.

$$t\left(x_p\right)=\frac{x_p}{L}$$

(22)

$$\alpha \left(x_p\right)=\frac{x_p}{f}$$

(23)

$$E={\left(x_p-x\left(t\left(x_p\right),\alpha \left(x_p\right),X,Y,Z\right)\right)}^2+{\left(y_p-y\left(t\left(x_p\right),\alpha \left(x_p\right),X,Y,Z\right)\right)}^2$$

(24)

where $x_p$ represents the image coordinate point, $L$ is the length of the film,$t\left(x_p\right)$ and $\alpha \left(x_p\right)$ are the scan time and rotation angle of the instantaneous strip image at $x_p$, respectively. $\left(X,Y,Z\right)$ are the ground point coordinates, and $x\left(t\left(x_p\right),X,Y,Z\right)$ and $y\left(t\left(x_p\right),\alpha \left(x_p\right),X,Y,Z\right)$ are the image coordinates calculated based on the time $t\left(x_p\right)$ and ground point $\left(X,Y,Z\right)$. To find the image coordinates that minimize the objective function $E$, this study adopts an iterative approach to update the exterior orientation elements as well as the image coordinates.

The specific steps of the orthorectification process are as follows:

(1)

We create a grid for the orthorectified image based on the coverage range of the KH-4B image and the desired resolution $s$, and interpolate the elevation values $Z$ for the grid points using DEM.

(2)

We initially set $t$ to 0.5, $x_p$ to half of the film length and $y_p$ to 0 for each ground point.

(3)

We first calculate $\left\{X_{st},Y_{st},Z_{st},{\omega }_t,{\phi }_t,{\kappa }_t\right\}$ and $\alpha$ based on $t$ and the solved 14 parameters. Then, we calculate the $x\left(t\left(x_p\right),\alpha \left(x_p\right),X,Y,Z\right)$ and $y\left(t\left(x_p\right),\alpha \left(x_p\right),X,Y,Z\right)$ of Equation (26) according to Equations (20) and (21). Finally, we adjust $t$ according to $x\left(t\left(x_p\right),\alpha \left(x_p\right),X,Y,Z\right)$. Repeat this step until $E$ is minimized. We summarize the process in Algorithm 1.

(4)

We interpolate the grayscale values of $\left(x_p,y_p\right)$ on the KH-4 image and assign them to the orthorectified image.

Algorithm 1: The specific steps of orthorectification

Input: ground point coordinates $\left(X, Y, Z\right)$; solved 14 parameters: $X_{s0}$,$Y_{s0}$,$Z_{s0}$,${\omega }_0$,${\phi }_0$,${\phi }_0$, ${\kappa }_0$,$X_{s1}$,$Y_{s1}$,$Z_{s1}$,${\omega }_1$,${\phi }_1$,${\kappa }_1$,$P$,$f$; film length: $L$.

Output: $\left(x_p, y_p\right)$.

$\mathrm{Initialization}: x_0$←L⁄2; $y_0$←0; $t_1$←0.5;$i$←1.

2. While (E < 1) or (i < 50) do

$X_{st}{}^i,Y_{st}{}^i,Z_{st}{}^i,{\omega }_t{}^i,{\phi }_t{}^i,{\kappa }_t{}^i,{\alpha }^i$ ←$t_i$

$x_i, y_i$←$f(X_{st}{}^i,Y_{st}{}^i,Z_{st}{}^i,{\omega }_t{}^i,{\phi }_t{}^i,{\kappa }_t{}^i,{\alpha }^i,P,f, X,Y,Z)$//$f(\ast )$ is the Equations (20) and (21)

$E={\left(x_i-x_{i-1}\right)}^2+{\left(y_i-y_{i-1}\right)}^2$

$t_{i+1}$←$t_i$; $x_{i+1}$, $y_{i+1}$←$x_i, y_i; x_p, y_p$←$x_i, y_i$

$i$←$i+1$

End

3. $\mathrm{Return}: x_p, y_p$

4. Extraction of Generalized Control Information

Differences in satellite orbit, sensors, and acquisition times result in radiometric, rotational, and scale distortions and changes in ground features between KH-4B images and reference images. To address this issue, this section proposes a robust algorithm for extracting generalized control information.

4.1. Feature Matching of the Corona Image and the Reference Image

4.1.1. Multiscale GU-FAST Feature Detection

To address radiometric distortion and feature point clustering, a method called GU-FAST is proposed. Specifically, GU-FAST first detects edges using the Sobel [31] operator and then applies the FAST algorithm with a low threshold to extract $N$ corner points ($N$ > $2M, M$ is the number of feature points to be detected), which are sorted based on the Harris [32] score. Next, within a range of ${\left(wh/\left(4M\right)\right)}^{-2}$, where $w$ and $h$ are the width and height of the image, respectively, GU-FAST searches for neighboring points and removes them from the set. Finally, the top $M$ key points with the largest responses are selected. To handle scale distortion, a scale space is constructed based on [33], and multiscale GU-FAST corner points are applied.

4.1.2. Rotation-Invariant Feature Description

Image rotation and non-linear radiometric distortion pose inevitable challenges in the matching of KH-4B images and references with non-linear intensity change, considering that the traditional feature description methods cannot handle non-linear intensity change and are sensitive to image rotation. To address this, a feature descriptor based on multi-directional features is proposed, which utilizes a multiscale, multi-directional Log–Gabor filter to construct multi-directional structural features (MR).

$$MR\left(x,y,o\right)={{\displaystyle \sum }}_s{A}_{so}\left(x,y\right)$$

(25)

$$Norm\left(x,y\right)=Sqrt\left({{\displaystyle \sum }}_{o}MR{\left(x,y,o\right)}^2\right)$$

(26)

where $A_{so}$ represents the filter feature of Log–Gabor, where $s$ and $o$ denote the scale and orientation of the Log–Gabor filter, respectively. $Norm\left(x,y\right)$ represents norm values.

A. Primary Orientation Estimation

Since the initial orientation of the Log–Gabor filter is fixed, the order of MR layers is highly sensitive to rotation distortion. Towards this, a primary orientation estimation algorithm based on the weighted norm feature of multiple-directional filtering is proposed. The specific algorithm follows the steps below:

(1)

Extract the norm values of a circular area around the feature point and apply Gaussian weighting to the area.

(2)

Identify evenly distributed multiple sectors with same-size overlapping regions within the circular area, as shown in Figure 4a. Specifically, we randomly create the first sector with a size of ${\theta }_2$ degrees, then rotate the sector sequentially by ${\theta }_1$ degrees clockwise. The adjacent sectors will have an overlapping of $\left({\theta }_2-{\theta }_1\right)$ angles. In this study, we set ${\theta }_1$ and ${\theta }_2$ as 5 and 30, respectively, obtaining 72 sectors in total.

(3)

Calculate the sum of weighted norms of all pixels within a sector for all for each sector.

(4)

Find the sector with the largest norm value and take the orientation of the central axis corresponding to this sector as the primary orientation according to the following equations.

Figure 4. The Estimation process of primary orientation with ${\theta }_1=5$, ${\theta }_2=30$. (a) computes weighted norm sums for pixels in each sector, while (b) identifies the sector with the highest norm value and assigns its central axis orientation as the primary orientation.

$$T=(N_{max}-1) \ast {\theta }_1+\frac{{\theta }_2}{2}$$

(27)

$$PO=\{\begin{array}{cc}\hfill T,& T<360\hfill \\ \hfill T-360,& T\ge 360\hfill \end{array}$$

(28)

where $N_{max}$ is the index of the sector region with the maximum norm value, $PO$ is the primary orientation within the range of [0°, 180°).

Additionally, the orientation of the central axis corresponding to the sector with the second largest norm is taken as the secondary primary orientation if the value exceeds 70% of the maximum norm value. We also build a feature descriptor with the secondary primary orientation.

B. Feature Descriptor Construction

Note that the primary orientation of each feature point has been obtained, and the order of the layers of the MR feature is adjusted according to the primary orientation, as shown in Table 3. Furthermore, because of the symmetry in the multi-directional filters, the order of each layer in MR remains consistent with the primary orientation of θ and θ + 180 degrees. For example, the third layer is moved to the first layer if the primary orientation is within the intervals 50–75° and 230–255°.

Table 3. The relationship between the layers order of MR and primary orientation.

Primary Orientation	The Order of Layers of MR
350–15°, 170–195°	{0°, 30°, 60°, 90°, 120°, 150°}
20–45°, 200–225°	{30°, 60°, 90°, 120°, 150°, 0°}
50–75°, 230–255°	{60°, 90°, 120°, 150°, 0°, 30°}
80–105°, 260–285°	{90°, 120°, 150°, 0°, 30°, 60°}
110–135°, 290–315°	{120°, 150°, 0°, 30°, 60°, 90°}
140–165°, 320–345°	{150°, 0°, 30°, 60°, 90°, 120°}

After that, the process of feature description is described as follows. First, multiple sampling points (12 directions, 3 concentric circles) are determined within the neighborhood of the feature point, as shown in Figure 5a. Second, for each sampling point, the multi-directional filter features (MR) within the circular neighborhood are weighted and summed using a Gaussian kernel, resulting in an o-dimensional sampling vector (Figure 5b). Finally, as shown in Figure 5c, starting from the primary orientation, the sampling vectors of each sampling point are concatenated clockwise to form a complete feature descriptor.

Figure 5. The pipeline of feature description. (a) shows sampling point distribution and numbering; (b) illustrates the construction of the sampling vector for point (3,1); (c) demonstrates feature vector construction by concatenating sampling vectors.

4.1.3. Feature Matching

We first perform pairwise matching using the nearest neighbor distance ratio [17] (NNDR) for each level of the image pyramid to obtain initial matches. Then, the mismatches are removed using the Forward Selection and Consistency [34] (FSC) method. Subsequently, the matches of all pyramid images are aggregated, and the FSC method is conducted on the fused match set. As a result, the final feature-matching results are obtained, and the transformation between the image pair is estimated.

4.2. Pyramid-Wise Template Matching

To further improve the matching performance, a pyramid-wise template matching strategy is applied. First, the reference image is resampled based on the estimated transformation matrix. Then, two image pyramids are constructed for the reference image and the KH-4B image, respectively. Finally, a template matching algorithm called CFOG [25] is employed for precise matching at each layer of pyramid images. The detailed process is as follows.

• At the top level of the pyramid, the corner features of the KH-4B image gradient map are extracted using the GU-FAST algorithm.
• The corner points detected on the KH-4B image are mapped to the reference image based on the transformation matrix, and template matching is performed using the CFOG [25].
• The matches are mapped to the next level of the image pyramid based on the resolution difference of different levels of the pyramid.
• The process 1–3 is repeated until the original resolution is reached.

4.3. Multi-Threshold Matching Enhancement

Local changes in the scene are an inevitable problem in matching multi-temporal remote sensing images, which can cause template matching to fall into local optima, producing erroneous matches. This significantly reduces the accuracy of control information and may even lead to failure in image orientation. Traditional match filtering methods struggle to eliminate these unreliable matches due to the following reasons: (1) The global reprojection error thresholds tend to be large due to the complex imaging model of KH-4B images, even when the matching points are completely correct. (2) Setting similarity thresholds becomes challenging due to non-linear radiometric distortions. (3) The panoramic camera model fails to converge when relatively low-accuracy generalized control points (16–32 m) from low-resolution layers in the image pyramid are used.

Aiming at eliminating the incorrect matches accurately, this study proposes a multi-threshold matching enhancement strategy (MTE) based on the scale and rotation change in a group of local feature points. Upon jointly considering a wide range of feature points, the unreliable points located in the change areas can be effectively removed, and more correct matches are found. The specific steps are as follows.

For a matching pair $P_k$ in the KH-4B image and $P_r$ in the reference image, we detect feature points with the FAST operator in a local area (~500 pixels) around $P_k$, a group of feature points can be obtained. Considering the scale change and rotation between the KH-4B image and the reference image have been roughly eliminated, the offsets of the newly detected points and $P_k$ can be directly applied to $P_r$ to predict the corresponding points of the detected points on the reference images. Specifically, for a new feature point Pi, its initial corresponding point can be calculated as $P_r-P_k+P_i$. After that, we use template matching to optimize the initial matching points and estimate a local transformation matrix using the refined matches. Based on the estimated transformation matrix, the scale change and rotation angle in the local area can be calculated. We set scale and rotation thresholds at 20% and 5°, respectively, based on the largest possible differences between KH-4B and reference imagery. If differences in scale and rotation are within these limits, we consider the local structures to be unchanged, and the new matches are retained; otherwise, the land cover has changed, and the new matches are discarded.

4.4. Model-Guided Matching

After obtaining sufficient relatively high-accuracy matches from the above matching process, we calculate the 14 parameters using the image model described in Section 3.2 and orthorectify the KH-4B image. Until now, notable geometric distortions caused by the intricate image process have been significantly reduced. We rematch the orthorectified image and the reference image to improve the accuracy. As shown in Figure 6, we first retain the previously obtained matching points in the reference image as feature points and discard the previous corresponding points on the KH-4B image. Second, we project the feature points to the KH-4B image with the calculated imaging parameters. Finally, we take the feature points as input and employ template matching on the reference image and coarsely orthorectified image to refine the matching results. The projected point on the KH-4B image in the second step and the adjusted feature point on the reference image in the last step are taken as a pair of generalized control points.

Figure 6. Model-guided image matching. First, translate the reference image’s feature points to the panoramic image. Then, reapply the template matching between the orthorectified local KH image block and the reference image. Finally, utilize these matching results as generalized control points.

5. Experiments and Results

To thoroughly evaluate the accuracy and generalization of 2OC, KH-4B images (listed in Table 4) from different locations with diverse terrain features and complex distortions were used as validation data. The evaluation was conducted by quantifying the accuracy of orientation and orthorectification.

Table 4. The details of multi-region Corona images and reference images.

KH-4A/B Image					Reference Image		Distortion
Region		Serial Number	Geomorphic Type	Resolution	Source	Resolution
United States	Vermont state	DS11161030DF009-10	Forest, Grassland, Lakes	~1.8	Google Earth	2.3	Land cover change, Radiometric distortion, Low/Repetitive texture, Image noise, Scale distortion, Rotation distortion, Panoramic geometry, Cloud occlusion
	Arizona state	DS11162161DA012-14	Plateau, Basin, Plain	~1.8	Google Earth	4.2
Russia	Khanty-Mansi	DS11102201DA033-34	Plain, Lakes	~1.8	Bing	4.2
China	Beijing	DS11011069DF089-94	Plain, Mountains	~1.8	Google Earth	4.2
	Chongqing	DS11142119DF045-49	Hills, Mountains	~1.8	Google Earth	2.3
	Gansu	DS11162297DA009-10	Loess Plateau	~1.8	Google Earth	4.2
	Qinghai–Tibet Plateau	DS11122265DA091-92	Glacier, Plateau	~1.8	Google Earth	4.2
Ethiopia	South West Shewa	DS11022203DA071-73	Plateau	~1.8	ArcGIS	4.2
Burkina Faso	Centre-sub	DS10451058DF045	Plateau	~2.75	ArcGIS	4.2

5.1. The Accuracy of Orientation

We first assess the accuracy of image orientation, which was evaluated using residual orientation and pose parameters. All obtained correspondences between the KH image and the reference orthophoto are used as generalized control points. The calculation of root mean square error (RMSE) of generalized control points is as follows:

$$m=\sqrt{\frac{{\sum }_i\left({\left(x_i-P\left(x_i\right)\right)}^2+{\left(y_i-P\left(y_i\right)\right)}^2\right)}{n}}$$

(29)

where $(x_i,y_i)$ is the image coordinate of the generalized control point, $P\left(x_i\right),P\left(y_i\right)$ are the image coordinates calculated using the panoramic camera model, $n$ is the number of generalized control points, and $m$ is the RMSE of residual. In Figure 7, the generalized control points required for DS1101-1069DF090b image orientation are depicted and extracted using the 2OC method.

Figure 7. The matches, namely the generalized control points, between the DS1101-1069DF090b image (c) and the reference Google orthophoto image (d), and the green dots indicate the control points, and the red numbers indicate the point numbers. (a,b,e,f) are the corresponding zoomed-in areas in (c,d).

Figure 7 shows the generalized control points required for DS1101-1069DF090b image orientation, extracted using 2OC. Overall, the 2OC approach demonstrates advantages in terms of both generalized control point quantity and distribution that manual extraction methods cannot achieve. In the zoomed-in view, the precision of these generalized control points has reached a level comparable to human recognition accuracy. The experimental results in Table 5 indicate that: (1) The overall accuracy is better than 2 pixels, with significantly higher accuracy in the central image blocks compared to the edge image blocks. This may be attributed to severe panoramic distortion and image deformation at the edges. (2) Model-guided matching effectively reduces the impact of panoramic distortion on the accuracy of control information, improving the orientation accuracy by 30% to 45%. For comparison with state-of-the-art work [28], we conducted experiments on the DS1117-2071DF008 dataset. The results show that the generalized control points extracted by 2OC outperform [28] in terms of quantity and distribution. The orientation accuracies are as follows: a (1.54), b (1.17), c (1.3), d (1.56), which are better than 1.94 of [28].

Table 5. The RMSE of orientation.

KH-4B Scenes		RMSE (Pixels)
		First Stage	Second Stage
DS1101-1069DF089	a	3.50	1.9
	b	2.1	1.2
	c	2.1	1.19
	d	2.3	1.3
DS1101-1069DF090	a	2.88	1.9
	b	2	1.2
	c	1.9	1.3
	d	2.4	1.4

Furthermore, to verify the generalization capability, we applied 2OC to the KH-4B images in Table 5 and presented the orthorectified images and mean errors of orientation in Figure 8. The overall accuracy is better than 2 pixels. The Gansu Province, known for its Loess Plateau, exhibits characteristics of weak texture, repetitive patterns, and significant speckle noise. Beijing and Vermont, USA, represent cases with land cover changes and radiometric distortions. The snow-covered region in Tibet leads to overexposure and severe non-linear radiometric distortion. The KH-4B images in Burkina Faso exhibit considerable noise due to camera and terrain factors. The area near the Ob River in Russia has numerous small lakes, where frozen water bodies in the KH-4B images are overexposed. In contrast, the water bodies in the reference image are underexposed, resulting in severe radiometric distortion. Moreover, changes in the lake edges over a 50-year period are observed. Figure 9 provides a detailed demonstration of complex distortions, including non-linear radiometric distortion, land cover changes, image noise, weak texture, cloud cover, and repetitive patterns. These results demonstrate that 2OC exhibits high generalization capability and can handle KH-4B images with various terrain types and complex distortions.

Figure 8. The orientation accuracy in various areas. The red font in the yellow square represents the orientation mean square error of each panoramic image. (a) The orthophoto of Russia; (b) The orthophoto of Gansu, China; (c) The orthophoto of Beijing, China; (d) The orthophoto of Vermont, USA; (e) The partial map of Russia; (f) The partial map of USA; (g) The world map; (h) The China Map; (i) The orthophoto of Chongqing, China; (j) The orthophoto of Arizona, USA; (k) The Burkina Faso map; (l) The Ethiopia map; (m) The orthophoto of Burkina Faso; (n) The orthophoto of Ethiopia; (o) The orthophoto of the Qinghai–Tibet Plateau.

Figure 9. The registration checkboard of the KH-4B orthophoto and reference image with complex image contents, where the image with a red dot is the KH-4B Orthophoto. (a–c) are located at the Qinghai–Tibet Plateau, and Beijing, China, with large NID; (d,e) are located in America and Beijing with land cover change; (f,h) are both located in Burkina Faso with noises; (g) is located in Gansu province, China with few textures; (i) is located in Ethiopia with large cloud coverage; (j) is located in Chongqing, China with repetitive textures.

Table 6 shows detailed attitude parameters for the DS1101-1069DF090 dataset. The attitude parameters between sub-images vary due to different primary point coordinates but should follow certain patterns. Here, X represents the east–west direction, Y represents the north–south direction, and Z represents the plumb line direction. The true scan time for a panoramic image is approximately 0.36 s (84,000 pixels), so the scan time for sub-images is approximately 0.154 s (36,000 pixels)/0.103 s (24,000 pixels). However, in this experiment, the scan time for sub-images is normalized to 1. Therefore, Ys1 should be around ~1.2 ${\mathrm{km} \mathrm{s}}^{-1}$/0.8 ${\mathrm{km} \mathrm{s}}^{-1}$ (equivalent to approximately 77.7 ${\mathrm{km} \mathrm{s}}^{-1}$ for a true panoramic camera), with an orbital altitude of around 170 km. $\varphi 0$ represents the rotation around the Y-axis, which represents the camera’s scanning angle. $\omega 0$ represents the rotation around the X-axis and should be around −15 degrees. However, the experimental results show variations between −10° and −20°. This may be due to the satellite’s attitude not strictly aligning with the plumb line on the ground, resulting in a deviation in the scanning direction of the camera. $\kappa 0$ represents the rotation around the Z-axis (plumb line direction), and from the distribution of generalized control points in Figure 7d of the Google image, a deviation of approximately 10° can be observed. The P (vertical height difference) for a and d is larger than that of b and c, which may be attributed to more severe image deformation at the ends.

Table 6. The 14 parameters of the four sub-images of DS1101-1069DF090.

Parameter	a	b	c	d
$X_{s0}\left(m\right)$	4,564,318.00	4,564,053.000	4,564,089.500	4,563,850
$X_{s1}\left(m\right)$	473.785004	1010.521729	552.231567	1008.983704
$Y_{s0}\left(m\right)$	384,394.15625	386,881.375000	387,326.562500	391,244.68750
$Y_{s1}\left(m\right)$	−796.56854	−1383.837524	−1283.782349	−909.642029
$Z_{s0}\left(m\right)$	169,839.609375	170,634.375000	170,839.453125	169,477.796875
$Z_{s1}\left(m\right)$	−812.061096	−605.189026	−630.709961	−196.461411
$\omega 0\left(deg\right)$	−11.1166419	−13.7147451	−16.6276366	−19.6613908
$\omega 1\left(deg\right)$	0.07700553	0.0135791	−0.02675713	−0.2566851
$\varphi 0\left(deg\right)$	28.2159374	13.3082628	−4.44374614	−23.0435035
$\varphi 1\left(deg\right)$	0.19194086	0.35964561	0.37299553	−0.37213609
$\kappa 0\left(deg\right)$	−11.274399	−9.7904165	−9.5870738	−10.9570732
$\kappa 1\left(deg\right)$	−0.27398842	0.06073353	0.15630289	0.08623015
P	−0.014932	0.017280	0.006359	−0.001736
$f\left(m\right)$	0.6025	0.6028	0.609602	0.6029

5.2. The Accuracy of Orthorectification

In this section, we present the registration checkerboard image between DS1101-1069DF092c and the reference image, as shown in Figure 10, and test the accuracy of the orthophoto generated using the proposed model. Orthorectified images are closely related to attitude parameters, and their accuracy reflects the accuracy of the attitude parameters. For a comprehensive evaluation, two metrics, the mosaic error and the error of the generalized control points, are applied. Specifically, we evaluate the mosaic accuracy based on three aspects: standard deviation (SD), maximum value (Max), and mean value (Mean) of the differences between the coordinate match points located in the overlapping regions of adjacent sub-image blocks. Table 7 provides detailed information accounting for the mosaic accuracy between the orthorectified image blocks of DS1101-1069DF089 and DS1101-1069DF090.

Figure 10. The registration results of the KH-4B image, DS1101-1069DF092c, and the reference image. (a–c) show the zoomed-in registration results of unchanged mountains; (d) shows the zoomed-in registration result of changed rivers. (e) The registration checkboard of the KH-4B orthophoto and reference image. (f) shows the zoomed-in registration result of the changed mountains area. (g,h) show the zoomed-in registration results of the changed plain area, where rivers have transformed into farmland and roads have undergone alterations. (i) shows the zoomed-in registration result of partially unchanged mountainous regions where rivers have experienced minor changes.

Table 7. The detailed mosaic error of DS1101-1069DF089 and DS1101-1069DF090.

Corona Scenes		Evaluation Metrics	X (Pixel)	Y (Pixel)
DS1101-1069DF089	a-b	SD	1.2762	1.3553
		Max	3.7658	4.3370
		Mean	0.9806	1.1052
	b-c	SD	0.5611	0.7644
		Max	1.4606	1.7897
		Mean	0.4570	0.6427
	c-d	SD	0.8033	0.8411
		Max	2.8138	1.9079
		Mean	0.6324	0.6927
DS1101-1069DF090	a-b	SD	0.4865	0.5811
		Max	1.5006	1.5650
		Mean	0.3814	0.4690
	b-c	SD	0.4614	0.8178
		Max	1.5528	1.9974
		Mean	0.3712	0.6845
	c-d	SD	0.8547	0.5226
		Max	2.4385	1.7930
		Mean	0.6790	0.4120

According to Table 7, the mosaic errors in the orthorectified images are mostly within 1 pixel, with the maximum value ranging from 1 to 4 pixels and the average value being better than 1 pixel. The mosaic errors at the edges of the image blocks are larger than those in the middle, which is consistent with the image orientation accuracy. This indicates that the proposed 14-parameter panoramic camera model accurately reverts the panoramic imaging process and can be used for orthorectification.

To obtain accurate ground checkpoints, sub-images were first stitched into a complete orthorectified image using the recovered georeferenced information. Then, 43 ground checkpoints were manually selected. Figure 11 shows the detailed distribution of these checkpoints. The root mean square error (RMSE) of the ground checkpoints in the X-direction is 3.89 m, and in the Y-direction is 3.29 m. The average RMSE in the X-direction is 2.69 m, and in the Y-direction is 2.52 m.

Figure 11. The orthophoto of DS1101-1069DF089, along with 42 checkpoints marked with red-cross dots.

6. Discussion

This section provides a detailed discussion of three critical steps in 2OC that may significantly affect the orientation and orthorectification accuracy, which are the PG stripe correction accuracy, the stability of the feature matching algorithm (NIFT), and the robustness of multi-threshold matching enhancement (MTE) to resist complex distortions.

6.1. The Impact of PG Stripe Correction on Orientation Accuracy

As described in Section 3.2, estimating rotation and translation components using the overlapping regions between adjacent image blocks may be affected by local image deformations, leading to stitching errors and a decrease in orientation accuracy. To simultaneously avoid stitching errors and scanning errors, this study adopts a method of processing individual image blocks before stitching. This is because the rotation and translation components of the image blocks are compensated for in the attitude parameters, including the angular and linear elements.

As shown in Table 8, the improvement in accuracy due to PG stripe correction for compensating image distortion is not as significant as [28]. Therefore, we constructed the PG stripe curves for each sub-image and found that this is mainly because the PG stripe curves of the sub-images are almost linear (as shown in Figure 12, Figure 13 and Figure 14), similar to image rotation. Hence, they can be compensated for using the angular elements of the attitude parameters. However, in the stitched complete image (with rotation and translation), as depicted in Figure 15, the PG stripe becomes more intricate. This complexity is likely due to the estimation bias in the rotation component. Consequently, PG stripe correction proves to be significant for the approach that involves image stitching followed by correction [28].

Table 8. The orientation accuracy for the four sub-images of DS1101-1069DF089 with/without film deformation adjustment based on PG stripe.

KH-4B Scenes		Without PG	With PG
DS1101-1069DF089	a	2.1	1.9
	b	1.34	1.2
	c	1.29	1.19
	d	1.4	1.3
DS1101-1069DF090	a	2.2	1.9
	b	1.3	1.2
	c	1.4	1.3
	d	1.5	1.4

Figure 12. The PG stripe curves of the four sub-images, a, b, c, d, of DS1101-1069DF089.

Figure 13. The PG stripe curves of the four sub-images, a, b, c, d, of DS1101-1069DF090.

Figure 14. The PG stripe curves of sub-images, a, b, c, d, of DS1105-1071DF141.

Figure 15. The PG stripe curve of the stitched panoramic image DS1105-1071DF141.

6.2. The Evaluation of NIFT

To evaluate the performance of NIFT, we created a dataset consisting of 30 pairs of down-sampled KH-4B images and reference images (as shown in Table 9) and compared it with SIFT [17], LNIFT [27], and RIFT [26]. For the comparative methods, the default parameters provided by the authors are used. For NIFT, the scale factor, orientation factor, descriptor orientation, and the number of concentric circles were set to {4, 6, 12, 3}, respectively. Accuracy rate (SR) and the number of correct matching points (NCM) were used as evaluation metrics. SR can be calculated as follows:

$$SR=\frac{N_s}{N_t}$$

(30)

where $N_s$ is the number of successfully matched image pairs, $N_t$ is the total number of image pairs involved, and SR represents the robustness of the matching algorithm.

Table 9. The details of the dataset used in feature matching.

Size (Pixel)	Resolution (m)	Number	Difference
500 × 200~1400 × 600	64~128	30	Radiometric distortion, land cover change, scale/rotation.

Table 10 gives the detailed matching results for the four algorithms. Among them, SIFT, as a classical method, only achieved a correctness rate of 6.6%. LNIFT algorithm obtained a higher number of NCM with 108. However, its SR was only 3.3%. RIFT had the same NCM as LNIFT but achieved higher robustness, with an SR of 53%. In comparison, NIFT successfully matched all image pairs and achieved more than three times NCM compared with the other algorithms.

Table 10. The compared matching results.

NCM				SR
SIFT	LNIFT	RIFT	NIFT	SIFT	LNIFT	RIFT	NIFT
20	108	109	373	6.6	3.3	53.3	100

To further evaluate the robustness of NIFT against rotation, we selected two representative images from Dataset 3 and manually rotated the images in steps of 5° within the range of [0, 360], creating 73 pairs of images with different rotation distortions. The matching results in Figure 16 demonstrate that although the NCM fluctuates to some extent due to the multi-directional filtering features, the algorithm is still able to obtain more than 190 correct matching points at any angle.

Figure 16. The NCM curve of NIFT under various rotational distortions.

6.3. The Evaluation of MTE

To evaluate the effectiveness of the multi-threshold matching enhancement strategy (MTE), we designed a series of experiments. To be simple, the 2OC without any matching enhancement strategy is referred to as $2O{C}_a$; the 2OC using the strategy of eliminating incorrect points based on reprojection error is referred to as $2O{C}_b$, where the reprojection error threshold is adaptively adjusted using three times the mean error; the 2OC enhanced with MTE is referred to as $2O{C}_c$. All three methods were applied to the same image dataset, which consists of DS1101-1069DF090 Corona image, Google Earth orthophoto, and 30 m STRM DEM data of the same area.

The experimental results in Table 11 show that there is no significant difference between the three methods in pairs a and b of image blocks, with an accuracy difference of less than 1 pixel. This is because the a and b regions are mountainous areas with little variation in land features and only a few low-quality generalized control points. In Region C, $2O{C}_a$ and $2O{C}_b$ achieved an accuracy of 3–4 pixels, while $2O{C}_c$ had an error of only 1.3 pixels. This is mainly due to the presence of two rivers with significant changes in Region C, resulting in a number of low-quality generalized control points and a large number of matching errors in image orientation. In Region D, where land feature changes are the most pronounced and low-quality generalized control points dominate, both $2O{C}_a$ and $2O{C}_b$ failed, while $2O{C}_c$ achieved an error of 1.4 pixels. This indicates that: (1) Complex distortions in the images indeed generate a large number of low-reliability match points, severely affecting the accuracy of image orientation and even preventing proper convergence. (2) The multi-threshold matching enhancement strategy effectively reduces the proportion of low-quality match points in the overall set of match points by eliminating low-reliability match points and increasing high-reliability match points, resulting in the correct convergence of the panoramic camera model estimation.

Table 11. The RMSE of orientation for DS101-1069DF90 of three methods. a, b, c, and d represent the sub-image of DS101-1069DF90.

Method	The RMSE (Pixels)
	a	b	c	d
$2O{C}_a$	2.6	1.3	4.2	-
$2O{C}_b$	2.5	1.3	3.1	-
$2O{C}_c$	1.9	1.2	1.3	1.4

7. Conclusions

This study presents a method for orientation and rectification of Corona KH-4B images (referred to as 2OC). First, to eliminate the focal length and panoramic distortion in KH-4B images, a panoramic mathematical model with 14 parameters and a time-iterative orthorectification method are proposed. Second, to counter complex distortions (radiometric/geometric distortions, weak texture, land changes, etc.) and automatically extract control information, a robust generalized control information extraction algorithm is proposed. Specifically, this includes the Maximum Sector Norm-based Primary Direction Estimation (NIFT), the Multi-Threshold Matching Enhancement Strategy (MTE) based on local texture, scale, and orientation, and the Model-Guided Matching.

Next, the generalization of 2OC is validated using KH-4B images with diverse terrain features (plateaus, glaciers, plains, hills, basins, etc.) and complex distortions from different locations worldwide (USA, Russia, Ethiopia, Burkina Faso, China) (Table 5). The results demonstrate superior orientation accuracy of better than 2 pixels, orthorectification accuracy of approximately 1 pixel, and planimetric accuracy of better than 4 m. Furthermore, detailed ablation and comparative experiments are conducted on the proposed NIFT, MTE, and Model-Guided Matching modules, highlighting their significant contributions to the generalization of 2OC. For example, allowing temporal differences (50 years) between the reference image and KH-4B image, non-linear radiometric distortion, rotational distortion ([0, 360)), scale distortion (1:4), and local land changes. Additionally, the rationality of separately processing sub-images and the impact of PG stripe correction are analyzed in detail.

Finally, the application of 2OC is not limited to handling Corona KH-4B images. By replacing the image mathematical model, it can be applied to a wider range of historical remote sensing images with complex multi-source and multi-temporal differences. In future studies, we will (1) attempt to use the rational function model commonly used in the remote sensing field to rectify KH-4B images for handling other historical remote sensing images with significant temporal differences and (2) explore the combination of manual features and deep learning features to extract control information more rapidly.