A TIR-Visible Automatic Registration and Geometric Correction Method for SDGSAT-1 Thermal Infrared Image Based on Modified RIFT

Abstract

High-resolution thermal infrared (TIR) remote sensing images can more accurately retrieve land surface temperature and describe the spatial pattern of urban thermal environment. The Thermal Infrared Spectrometer (TIS), which has high spatial resolution among spaceborne thermal infrared sensors at present, and global data acquisition capability, is one of the sensors equipped in the SDGSAT-1. It is an important complement to the existing international mainstream satellites. In order to produce standard data products, rapidly and accurately, the automatic registration and geometric correction method needs to be developed. Unlike visible–visible image registration, thermal infrared images are blurred in edge details and have obvious non-linear radiometric differences from visible images, which make it challenging for the TIR-visible image registration task. To address these problems, homomorphic filtering is employed to enhance TIR image details and the modified RIFT algorithm is proposed to achieve TIR-visible image registration. Different from using MIM for feature description in RIFT, the proposed modified RIFT uses the novel binary pattern string to descriptor construction. With sufficient and uniformly distributed ground control points, the two-step orthorectification framework, from SDGSAT-1 TIS L1A image to L4 orthoimage, are proposed in this study. The first experiment, with six TIR-visible image pairs, captured in different landforms, is performed to verify the registration performance, and the result indicates that the homomorphic filtering and modified RIFT greatly increase the number of corresponding points. The second experiment, with one scene of an SDGSAT-1 TIS image, is executed to test the proposed orthorectification framework. Subsequently, 52 GCPs are selected manually to evaluate the orthorectification accuracy. The result indicates that the proposed orthorectification framework is helpful to improve the geometric accuracy and guarantee for the subsequent thermal infrared applications.

1. Introduction

The thermal infrared (TIR) bands are the significant channels for land surface temperature retrieval [1,2] and urban heat island effect monitoring [3,4,5], as well as croplands evapotranspiration studies [6,7,8]. In the past few decades, TIR remote sensing has been developed rapidly. The resolution of spaceborne TIR images and the global data acquisition capability of TIR satellites are improved continuously. SDGSAT-1, successfully launched on 5 November 2021, is the world’s first space scientific satellite dedicated to serving the U.N. 2030 Agenda for Sustainable Development Goals (SDGs). The Thermal Infrared Spectrometer (TIS) is one of the sensors equipped in the SDGSAT-1 [9]. TIS uses a long linear array whisk broom method, which combines both high spatial resolution and large width, with a spatial resolution of 30 m and a width of 300 km. It has high spatial resolution among the in-orbit spaceborne TIR imagers at present, and will be an important supplement to the existing earth observation TIR data. TIS has three channels, i.e., 8.0~10.5 μm, 10.3~11.3 μm and 11.5~12.5 μm.

Table 1. SDGSAT-1 TIS characteristics and the comparison of other TIR sensor.

Sensor	Platform	Wavelengths (μm)	Resolution (m)	Swath Width (km)	Altitude (km)	Imaging Method
TIS	SDGSAT-1	8.0~10.5, 10.3~11.3 11.5~12.5	30	300	505	Whisk broom
VIMS	GF-5	8.01~8.39, 8.42~8.83 10.3~11.3, 11.4~12.5	40	60	705	Push broom
IRS	HJ-1B	10.5~12.5	300	720	649	-
IRMSS	CBERS-04	10.4~12.5	80	120	778	Push broom
TIRS	Landsat8\9	10.6~11.2, 11.5~12.5	100	185	705	Push broom
ASTER	TERRA	8.13~8.48, 8.47~8.83, 8.93~9.28, 10.25~10.95 10.95~11.65	90	60	705	Push broom

lists the argument of TIS and the comparison of other TIR sensors in orbit.

In order to produce SDGSAT-1 TIS orthoimage products, rapidly and accurately, the automatic registration and geometric correction method needs to be developed. Considering that there are no TIR images with the same 30m resolution as the reference image, the TIR-visible registration method is proposed in this study. However, due to the different imaging mechanisms, there are significant non-linear radiometric differences (NRD) between TIR and visible images, which bring enormous challenges to the TIR-Visible image registration. Thus, it is necessary to develop an effective registration method for TIR and visible remote sensing images.

Generally speaking, traditional registration methods can be divided into two types, area-based methods and feature-based methods [10,11,12]. Area-based methods, using a template matching scheme, adopt the original pixel values of images to establish the similarity metric between image pairs, so as to determine the corresponding points. The common area-based methods include normalized cross-correlation (NCC) [13,14], mutual information (MI) [15,16] and phase correlation, based on Fourier transform [17,18,19]. This kind of method is robust to linear radiance differences and the translation changes of images, but they have no robustness to scale, rotation and non-linear radiance differences. The computation efficiency is low because of using the template matching strategy. Feature-based methods adopt the gradient information of images. Firstly, some stable and obvious feature points from the sensed image and the reference image are extracted by some feature extraction operators. Then, the image patches centered on the feature points are extracted and the feature descriptors of the features points are constructed, according to the gradient information of the image patches. Finally, the two points in the image pairs, having the closest distance of feature descriptors, are regarded as the corresponding points. Scale Invariant Feature Transform (SIFT) is the classical feature-based method [20,21]. In subsequent studies, scholars make a series of improvements to SIFT in computing efficiency and denoising, such as speeded-up robust features (SURF) [22,23], principal component analysis SIFT (PCA-SIFT) [24], SAR-SIFT [25] and OS-SIFT [26]. All these SIFT-like algorithms use the gradient to construct descriptors, which is sensitive to radiance changes. These algorithms are not robust to non-linear radiometric difference and achieve bad performance in multimodal remote sensing image registration, such as TIR-Visible registration.

At present, with the rapid development of deep learning in computing vision, learning-based registration methods have drawn much attention [27,28,29]. The main ideas of learning-based methods are to use deep convolution neural networks (CNN) to extract the feature points [30,31], or learn the descriptors’ construction pattern [32,33,34], or build the similarity metric function about descriptors [35,36]. These steps can also be joint to form an end-to-end architecture for image registration [37,38]. Compared with the classic registration methods, learning-based registration adopts the data-driven mechanisms to learn the registration strategy, including robustness to scale, rotation and non-linear radiance differences, but the drawbacks are also obvious. The training process needs enormous training samples and vast hyperparameters need to be tuned manually. The generalization of deep learning registration methods is unsatisfactory.

Since neither traditional ways nor learning-based methods can effectively address the problem of non-linear radiation difference in remote sensing image registration, another model, called phase congruency (PC) [39], is taken into account, gradually. This model can detect image features, such as points and lines with illumination and contrast invariance [40]. The histogram of orientated phase congruency (HOPC) extends the PC model and defines a novel similarity metric [41,42]. HOPC uses a Harris Operator to extract features in the sensed image, and then calculates the PC values of the sensed image and reference image. Subsequently, using the strategy of template matching, the points that get the highest scores of similarity metric, in each template, are selected as corresponding points. Due to the use of template matching, this algorithm is essentially an area-based method, which cannot address the scale and rotation changes. It needs to know the accurate geographic information and the offsets between sensed image and reference image, which must be within several pixels. If not, the cost of computation is very high. Radiation-invariant Feature Transform (RIFT) is another PC-extending model that combines features-based methods [43]. RIFT uses PC values instead of pixel values for feature points detection, which can extract enormous and repeatable points in both the sensed image and reference image. Then, a maximum index map (MIM) is proposed for feature description. RIFT is robust to non-linear radiance difference and rotation changes, and achieves excellent performance in the multimodal remote sensing image registration.

Comparing with visible–visible image registration, high-resolution TIR-visible image registration has three challenges: (1) there is an obvious nonlinear difference between the TIR image and visible image in gray scale, which is not conducive to the construction of descriptors and the selection of the metric function; (2) ground objects with a different spectrum in visibility may be the same or similar in temperature, resulting in unclear texture details and ambiguous image edges of thermal infrared images, which are not conducive to selecting stable and repeatable feature points; (3) TIR images are noisy, and TIR-Visible image registration is easily affected by noise. To address the problems mentioned above, and implement production for the Level4 (L4) orthorectification TIR images of SDGSAT-1, a modified RIFT algorithm for registration and the orthorectification framework for SDGSAT-1 TIS images are proposed in this study. The main three contributions of this study are summarized as follows:

• Homomorphic filtering is employed before the feature points extraction stage to denoise and enhance the texture details of TIR images.
• In the descriptors construction stage, a novel binary pattern string is proposed, which is more robust to NRD than the MIM of the original RIFT. The binary pattern string is able to express a log-Gabor convolution sequence, while reducing computational complexity.
• Two-step orthorectification method from Leve1A (L1A) to L4 is designed. For the first step, the L1A TIR image is orthorectified with RPC coefficients and DEM data. Then the orthorectified image is registered with the reference visible image using the modified RIFT. To get the ground control points (GCPs), the corresponding points in the orthorectified image are remapped back to the L1A image coordinates and the points in the reference visible image are mapped to geographic coordinates. Later, the RPC refinement is executed with GCPs. For the second step, the L1A TIR image is orthorectified again, with refined RPCs coefficients and DEM, resulting the L4 production.

This paper is organized as follows. Section 2 describes the TIR-visible image registration method and the orthorectification flow of the SDGSAT-1 TIS L1A image. Section 3 states two experimental designs and the results. Section 4 discusses the experiment results and Section 5 is the conclusion.

2. Methodology

The modified RIFT algorithm is described in detail in Section 2.1 and the framework of orthorectification for the SDGSAT-1 TIS image is outlined in Section 2.2.

2.1. Modified RIFT

Most of the rotation distortion of SDGSAT-1 TIS image can be removed by selecting a few GCPs manually or orthorectification with the initial RPC coefficients. The scale difference can be eliminated by down-sampling the visible reference image to the same resolution as TIR image. Thus, the modified RIFT algorithm only takes the non-linear radiometric difference into account.

2.1.1. Homomorphic Filtering

All the registration methods in computer vision are applied for the 8-bit nature images, while the SDGSAT-1 TIS images are 12-bit. It needs to compress the TIR images into 8-bit. Because TIR images represent the temperature information, the pixel values in the same scene are concentrated at a certain interval, which is inappropriate for a global linear stretch in gray. The global linear gray transform from 12-bit to 8-bit will lose a lot of detail, which makes it difficult for the feature detection later.

Homomorphic filtering is a non-linear filter, which can compress the gray range of the image in the frequency domain and improve the image contrast at the same time, and enhance the details in the dark area of the image without losing the details in the bright area [44,45,46]. Homomorphic filtering assumes that the image $\mathrm{f}\left(\mathrm{x},\mathrm{y}\right)$ is regarded as the product of the low-frequency incident component $\mathrm{i}\left(\mathrm{x},\mathrm{y}\right)$ and high-frequency reflection component $\mathrm{r}\left(\mathrm{x},\mathrm{y}\right)$. This is $\mathrm{f}\left(\mathrm{x},\mathrm{y}\right)= \mathrm{i}\left(\mathrm{x},\mathrm{y}\right)\mathrm{r}\left(\mathrm{x},\mathrm{y}\right)$, where $\left(\mathrm{x},\mathrm{y}\right)$ stands for image coordinates. A transfer function $\mathrm{H}\left(\mathrm{u}, \mathrm{v}\right)$ is applied for non-linear gray enhancement in the frequency domain, and the specific process is shown in Figure 1.

The transfer function $\mathrm{H}\left(\mathrm{u},\mathrm{v}\right)$ is expressed as:

$$\mathrm{H}\left(\mathrm{u},\mathrm{v}\right)=\left({\mathrm{r}}_{\mathrm{H}}-{\mathrm{r}}_{\mathrm{L}}\right)\left(1-{\mathrm{e}}^{-\mathrm{c}\frac{{\mathrm{D}}^2\left(\mathrm{u},\mathrm{v}\right)}{{\mathrm{D}}_0^2}}\right)+{\mathrm{r}}_{\mathrm{L}}$$

(1)

where $\mathrm{D}\left(\mathrm{u},\mathrm{v}\right)$ is the distance from the image coordinate point $\left(\mathrm{u},\mathrm{v}\right)$ to the image center in the frequency domain. ${\mathrm{r}}_{\mathrm{H}}$, ${\mathrm{r}}_{\mathrm{L}}$, ${\mathrm{D}}_0$ and c are the tuning parameters. When ${\mathrm{r}}_{\mathrm{L}}<1$ and ${\mathrm{r}}_{\mathrm{H}}>1$, the filter weakens the low-frequency part of the image and enhances the high-frequency part of the image. The final result is that the texture details of the image are enhanced while the gray scale range of the image is compressed. In this study, the parameters are set as ${\mathrm{r}}_{\mathrm{H}}=4$, ${\mathrm{r}}_{\mathrm{L}}=0.5$, and $c=4$. ${\mathrm{D}}_0$ is taken as a quarter of the width of the input image.

2.1.2. Feature Points Extraction

A large number of feature points, which are repeatable and evenly distributed in the sensed image and reference image, are selected during the feature points extraction process. Traditional extraction methods usually depend on image pixel values or gradient information, which are spatial domain information. These methods are sensitive to the illumination and extract little repeatable points in the TIR image and visible reference image. Except spatial domain information, frequency domain information, such as phase, can also be applied in feature points extraction. Phase information has high invariance to non-linear radiometric changes [47,48]. Phase congruency (PC) model is used for measuring the degree of consistency in the local phase information at different angles. The PC model at the specific orientation ${\mathsf{\theta }}_o$ is described as follows:

$$\mathrm{PC}\left(x,y;{\mathsf{\theta }}_o\right)=\frac{{\sum }_s{w}_o\left(x,y\right)\lfloor A_{so}\left(x,y\right){\mathsf{\Delta }\mathsf{\Phi }}_{so}\left(x,y\right)-T\rfloor }{{\sum }_s{A}_{so}\left(x,y\right)+\mathsf{\xi }}$$

(2)

where $\left(x,y\right)$ is the image coordinate; $w_o\left(x,y\right)$ is a weighting function; $\mathsf{\xi }$ is a small value to prevent denominator being zero; $T$ is the noise compensation term; $\lfloor \cdot \rfloor$ operator takes zero when the enclosed quantity is negative, and itself when the enclosed quantity is non-negative. $A_{so}\left(x,y\right){\mathsf{\Delta }\mathsf{\Phi }}_{so}\left(x,y\right)$ is defined as:

$$A_{so}\left(x,y\right){\mathsf{\Delta }\mathsf{\Phi }}_{so}\left(x,y\right)=\left(E_{so}\left(x,y\right){\overline{\mathsf{\varphi }}}_E\left(x,y\right)+O_{so}\left(x,y\right){\overline{\mathsf{\varphi }}}_O\left(x,y\right)\right)-\left|\left(E_{so}\left(x,y\right){\overline{\mathsf{\varphi }}}_O\left(x,y\right)-O_{so}\left(x,y\right){\overline{\mathsf{\varphi }}}_E\left(x,y\right)\right)\right|$$

(3)

where,

$${\overline{\mathsf{\varphi }}}_E\left(x,y\right)={\sum }_s{E}_{so}\left(x,y\right)/\mathrm{C}\left(x,y\right), {\overline{\mathsf{\varphi }}}_O\left(x,y\right)={\sum }_s{O}_{so}\left(x,y\right)/C\left(x,y\right)$$

(4)

$$A_{so}=\sqrt{E_{so}{\left(x,y\right)}^2+O_{so}{\left(x,y\right)}^2}$$

(5)

and,

$$\mathrm{C}\left(x,y\right)=\sqrt{{\left({{\displaystyle \sum }}_s{E}_{so}\left(x,y\right)\right)}^2+{\left({{\displaystyle \sum }}_s{O}_{so}\left(x,y\right)\right)}^2}$$

(6)

The $E_{so}\left(x,y\right)$ and $O_{so}\left(x,y\right)$ are the real part and imaginary part respectively of the image convolution result using 2D log-Gabor filter. The 2D log-Gabor filter, which is determined by the orientation $o$ and scale $s$, is defined as:

$$L\left(\mathsf{\rho },\mathsf{\theta };s,o\right)=exp\left(\frac{-{\left(\mathsf{\rho }-{\mathsf{\rho }}_s\right)}^2}{2{\mathsf{\sigma }}_{\mathsf{\rho }}^2}\right)exp\left(\frac{-{\left(\mathsf{\theta }-{\mathsf{\theta }}_o\right)}^2}{2{\mathsf{\sigma }}_{\mathsf{\theta }}^2}\right)$$

(7)

where $\left(\mathsf{\rho },\mathsf{\theta }\right)$ is the log-polar coordinate; $\left({\mathsf{\rho }}_s,{\mathsf{\theta }}_o\right)$ is the center frequency; ${\mathsf{\sigma }}_{\mathsf{\rho }}$ and ${\mathsf{\sigma }}_{\mathsf{\theta }}$ are the bandwidths in $\mathsf{\rho }$ and $\mathsf{\theta }$, respectively. The detailed calculation process can be referred to [43].

After obtaining the PC map with specific orientation according to Equation (2), the minimum moment $m_{\mathsf{\psi }}$ and maximum moment $M_{\mathsf{\psi }}$ of phase congruency covariance are calculated as follows:

$$m_{\mathsf{\psi }}=\frac{1}{2}\left(c+a-\sqrt{b^2+{\left(a-c\right)}^2}\right)$$

(8)

$$M_{\mathsf{\psi }}=\frac{1}{2}\left(c+a+\sqrt{b^2+{\left(a-c\right)}^2}\right)$$

(9)

where

$$a={\displaystyle \sum }_o{\left(PC\left({\mathsf{\theta }}_o\right)cos\left({\mathsf{\theta }}_o\right)\right)}^2$$

(10)

$$b=2{\displaystyle \sum }_o\left(PC\left({\mathsf{\theta }}_o\right)cos\left({\mathsf{\theta }}_o\right)\right)\left(PC\left({\mathsf{\theta }}_o\right)sin\left({\mathsf{\theta }}_o\right)\right)$$

(11)

$$c={\displaystyle \sum }_o{\left(PC\left({\mathsf{\theta }}_o\right)sin\left({\mathsf{\theta }}_o\right)\right)}^2$$

(12)

In this study, 8 orientations are applied and ${\mathsf{\theta }}_o$ gets the value in the list {0°, 22.5°, 45°, 67.5°, 90°, 112.5°, 135°, 157.5°} in turn. In PC model, minimum moment $m_{\psi }$ can be used as a measure of corner strength and maximum moment $M_{\psi }$ can be used as measure of edge strength. They are insensitive to the non-linear radiometric changes. Then the FAST feature detector [49] is executed on the minimum and maximum moment maps to extract repeatable points in both TIR image and visible image.

2.1.3. Feature Description

After extracting points in both sensed and reference images, descriptors for each point are constructed to perform feature matching. Descriptors need to be distinguishable. The distance between two description vectors for corresponding points is as close as possible, while is as far as possible for unrelated points. Different from using maximum index map (MIM) for feature description in RIFT, a novel binary pattern string is proposed in this study.

The binary pattern string is constructed via 8 orientations log-Gabor convolution sequence, obtained in the feature points extraction stage. For each orientation $o$, the amplitudes of all $s$ scales are summed up to get the log-Gabor convolution layer $A_o\left(x,y\right)$:

$$A_o\left(x,y\right)={\displaystyle \sum }_s{A}_{so}\left(x,y\right)$$

(13)

Thus, for each position $\left(x,y\right)$ in image coordinate, 8 convolution values can be obtained ordered by the orientations list mentioned above. In RIFT, the orientation index of the maximum convolution value is selected to construct the MIM map. The shortcoming of this method is that the functional relationship between the orientation and the convolution cannot be completely described. In this study, the orientation indexes of maximum and minimum value are selected and the numerical relationship between adjacent orientations are taken into account. The 0 or 1 sign is defined as follows:

$$sign\left(o\right)=\{\begin{array}{c}1,\hspace{1em} A_o>A_{o+1}\\ 0,\hspace{1em} A_o<A_{o+1}\end{array} where \left(o\in \left\{0,1,2,3,4,5,6,7\right\} \mathrm{and} \mathrm{when} \mathrm{o}=7, \mathrm{take} \mathrm{o}+1=0\right)$$

(14)

Then an 8-bit binary string is generated for each pixel. In order to be consistent with the binary string, the indexes of the maximum and minimum convolution, for which the value is the integer from 0 to 7, are also converted to binary number. Both indexes are converted into 3-bit binary. Finally, a 14-bit binary string is generated for each pixel and a binary data cube is constructed with 14 channels. Theoretically, the more orientations taken, the more details the relationship between orientations and convolution achieves. However, more orientations will increase the computation so 8 orientations are considered in this study.

After obtaining the binary pattern string, the distribution histogram technique similar to RIFT is applied for feature description. In detail, for each feature point, an image patch is extracted with $J\times J\times 14$ pixels centered at the point. Then the patch is divided into $6\times 6$ sub-grids. For each sub-grid, sum up the binary value in every channels. The description vector is obtained by concatenating the summation of all the sub-grid and the dimension of the vector is $6\times 6\times 14$. Then the vector is compressed and normalized. Finally, the descriptor with 504 dimension is obtained. In this study, the value of $J$ is 96.

2.1.4. Feature Matching and Outliers Removal

After obtaining the descriptors of TIR and visible image, the Fast Library for Approximate Nearest Neighbors (FLANN) [50] is adopted for feature matching. The second-order transformation function is used for image transform and outliers removal with Random Sample Consensus (RANSAC) [51]. The second order transformation is defined as follows:

$$\{\begin{array}{c}{x}^{\prime }=a_0+a_{1}x+a_{2}y+a_{3}xy+a_4{x}^2+a_5{y}^2\\ y^{\prime }=b_0+b_{1}x+b_{2}y+b_{3}xy+b_4{x}^2+b_5{y}^2\end{array}$$

(15)

where $\left(x,y\right)$ is the coordinate of sensed image and $\left(x^{\prime },y^{\prime }\right)$ is the coordinate of reference image; $a_0,\dots ,a_5,b_0,\dots ,b_5$ are the transform coefficients that need to be fitted.

Finally, the correct corresponding points are obtained and used for fitting the transform coefficients again. The second-order image transformation is executed to align the TIR and visible image.

The total process of modified RIFT for TIR-VIS image registration is shown in Figure 2:

2.2. Orthorectification Framework for SDGSAT-1 TIS Image

SDGSAT-1 TIS uses the long linear array whisk broom method to capture the thermal infrared image with 300-km width and 30-m spatial resolution. An image with $1987\times 10,000$ pixels is captured in every scan. Combining with the sensor and ephemeris parameters, the RPC coefficients are derived. Then the L1A image products are generated. In this study, an orthorectification framework for SDGSAT-1 TIS image from L1A to L4 is proposed.

Firstly, the L1A TIR image is orthorectified with RPC coefficients and DEM data. Then the orthorectified image is registered with the reference visible image using the modified RIFT algorithm. The TIR image has 3 thermal infrared bands and they have been aligned in the L1A product. Therefore, only the B1 band (8.0~10.5 μm), which is closer to the visible band than the others, is used for registration.

Lots of corresponding points are obtained after the registration. However, the registration only records the corresponding points’ row and column index with respect to the orthorectified image and reference image, which is not consistent with the ground control points (GCPs) format. Thus, the points in the orthorectified image are remapped back to the L1A image coordinate space by the RPC inverse transforming. Further, the points in the reference visible image are mapped to geographic coordinate. The DEM value for each GCP can be retrieve by the geographic coordinate. Each GCP is recorded as its row and column index in L1A image coordinate space, the latitude and longitude vales, as well as DEM value. Then the RPC refinement is executed with GCPs.

Finally, the L1A TIR image is orthorectified again with refined PRC coefficients and DEM to generate the L4 production. The orthorectification process is shown in Figure 3.

3. Experiment and Results

The two experiment designs, regarding the modified RIFT registration and SDGSAT-1 TIS image orthorectification, respectively, are described in Section 3.1, and the experiment results are displayed in Section 3.2.

3.1. Experiment Design

The performances of the proposed modified RIFT algorithm and orthorectification framework are tested in the following two experiments.

3.1.1. Experiment for Modified RIFT

Six image pairs, TIR images of SDGSAT-1 and visible images of Landsat and GF1-WFV, are used for this experiment. Each image has $1000\times 1000$ pixels. The TIR and Landsat visible images have 30-m spatial resolution. The GF1-WFV visible images have 16-m spatial resolution and are resampled to 30 m. The reason for using Landsat and GF1-WFV as a reference image is that they have global data acquisition capability and are free open sharing. These test image pairs are captured on different days and have different landforms in China. Each image pair has more than 50 percent area overlap. The TIR images have been coarsely registered by several manual GCPs to obtain the georeferenced coordinate. The visible images are orthoimages. All the test image pairs, and their detailed information, are shown in Figure 4 and

Table 2. The detailed information about the test image pairs.

Image Pair		Acquired Date	Type	Position
Pair1	TIR (SDGSAT-1) Visible (Landsat-8)	15 December 2021 18 November 2021	Farmland	Aksu, Xinjiang, China
Pair2	TIR (SDGSAT-1) Visible (Landsat-8)	15 December 2021 13 December 2021	Plateau	Aksu, Xinjiang, China
Pair3	TIR (SDGSAT-1) Visible (Landsat-8)	14 December 2021 12 December 2021	Lake	Ulan UL Lake, Qinghai, China
Pair4	TIR (SDGSAT-1) Visible (Landsat-8)	20 December 2021 18 September 2021	Plain	the Northeast Plain, China
Pair5	TIR (SDGSAT-1) Visible (GF1-WFV)	2 January 2022 26 December 2021	Mountain	Taiyuan, Shanxi, China
Pair6	TIR (SDGSAT-1) Visible (GF1-WFV)	2 January 2022 16 December 2021	City	Taiyuan, Shanxi, China

.

3.1.2. Experiment for Orthorectification of SDGSAT-1 TIS

One scene of the SDGSAT-1 TIS image, with $1987\times 10,000$ pixels, is used for this experiment. The TIR image is the L1A product with RPC coefficients. It was captured on 6 January 2022, in Taiyuan, China. The visible image of Landsat 8, with 30-m spatial resolution, is an orthoimage and is used as the reference base image. The SRTM data, with 30-m spatial resolution, is used as the DEM. All the Landsat 8 and SRTM data are downloaded from https://earthexplorer.usgs.gov (12 March 2022) and mosaiced by the ENVI software to generate the corresponding range with the TIR image. All the test data are shown in Figure 5.

In order to improve the computation efficiency, the L1A TIR image is divided into a $500\times 500$ pixels patch, which has, totally, $4\times 20$ patches. Considering the georeferenced offset exists, the visible patch is set to $800\times 800$ pixels. Each TIR patch is registered with the corresponding visible image patch. Only five corresponding points are selected randomly in each patch. Totally, 400 GCPs are used for refining the RPC coefficients. The RPCs’ refinement and orthorectification are executed with the ENVI RPC Orthorectification Workflow toolbox.

3.2. Result

In order to show the registration results more clearly, the TIR images are displayed with the homomorphic filtering in the following sections.

3.2.1. Results of Modified RIFT

As shown in Figure 6, the modified RIFT can effectively find enough corresponding points between the TIR and visible images, in the situation of different sensed time, different landforms and different sensors. The images in pair3 and pair4 have relatively large georeferenced offset between TIR and visible, but the modified RIFT can still find corresponding points in the overlap areas. The visible images come from different sensors, Landsat8 and GF1-WFV, and fair registration consequences are achieved. The modified RIFT is robust enough in the TIR-visible registration task for SDASAT-1.

3.2.2. Results of Orthorectification

As shown in Figure 7, some patch pairs’ registration results are displayed and all GCPs, extracted after the registration, are plotted on the original L1A TIR image. Each GCP contains the index of column and row, latitude and longitude coordinates, and DEM value. The L4 TIR image is shown in Figure 8, overlaying with the reference image to display the orthorectification result. Combining with the modified RIFT, the orthorectification framework proposed in this study can extract GCPs automatically from SDASAT-1 L1A TIR products and achieve L4 orthoimage production effectively.

4. Discussion

In this section, the improved performances of the modified RIFT are discussed and the registration and orthorectification accuracy are analyzed.

4.1. Performances of the Homomorphic Filtering

The traditional registration algorithms in computer vision are suitable for an 8-bit nature image, while the SDGSAT-1 TIS images are 12-bit. Bit-depth transform or gray linear stretch, from 12-bit to 8-bit, may compress image details, which is bad for the descriptor construction during the registration. In order to preserve the image detailed information, the homomorphic filtering is proposed in this study. The contrast test is performed as follows.

Figure 9 displays the TIR images’ gray transformation by using bit-depth transform, gray linear stretch and homomorphic filtering, respectively. It is quite obvious that the homomorphic filtering can enhance the image contrast and keep the image edge details, when transforming TIR images from 12-bit to 8-bit. Bit-depth transform compresses the effective grayscale range of the image and reduces the image contrast. Gray linear stretch keeps the effective grayscale range, but the edge details are blurrier than homomorphic filtering. Figure 10 shows the registration result by using the modified RIFT, with three different gray transformations. The correct corresponding points are counted. The homomorphic filtering gets more corresponding points than the other two gray transformation methods, which indicates that it is necessary to execute homomorphic filtering for the TIR image before registration.

4.2. Performances of Modified RIFT

The comparison test is performed to establish the potentially improved performances of modified RIFT. Four registration methods, which are invented for the cross-modality remote sensing image registration, i.e., MI, HOPC, CMM-Net, RIFT, are used as the comparative experiment with the modified RIFT proposed in this study. All these methods include the traditional methods and the state-of-the-art methods. They are applied for the six test image pairs. Figure 11 shows the registration results. The number of correct corresponding points is considered as the evaluation index. The statistical results are shown in

Table 3. The number of correct corresponding points of different registration methods.

	MI	HOPC	CMM-Net	RIFT	Modified RIFT
Pair1	8	54	35	82	80
Pair2	14	30	44	96	110
Pair3	28	0	14	52	70
Pair4	16	30	73	89	106
Pair5	13	90	122	260	265
Pair6	28	23	70	106	138

.

MI is the traditional area-based method. Because it is easy to calculate, the MI method has been integrated into the image-processing software, such as ENVI. It is more robust to non-linear radiometric differences than other traditional area-based methods. In this test, MI can get some corresponding points, but not enough. Due to using the template matching scheme, the offset between the sensed image and the reference image must be within several pixels. If the offset is too large, such as pair3, the search area needs to be widened, which uses more computation time. HOPC is one of the state-of-the-art area-based methods that combines with the PC model. It performs better than the MI, except for pair3. The HOPC needs to know the accurate geographic information and is invalid when the offset between the sensed and reference image is too large. CMM-Net [52] is one of the state-of-the-art deep learning registration methods. It uses D2-Net as the feature extractor, to extract corresponding points efficiently and construct robust descriptors. It achieves an acceptable result in six test pairs. However, CMM-Net needs vast TIR-visible image pairs as training data and lacks generalization. RIFT is the feature-based registration method that combines with the PC model. It achieves a satisfying result in all the test pairs. Modified RIFT gets better performance than RIFT because of adopting the homomorphic filtering and the binary pattern string construction of the descriptor. The modified RIFT has high generalization in different landforms.

In terms of time efficiency, modified RIFT also has satisfying performances. Although CMM-Net spends the least time in the test, it needs a large amount of time to train the model. The processing time of MI and HOPC is not balanced in the six test pairs. When the offset is large, the search radius of template matching needs to increase, which results in more processing time. The time of modified RIFT is slightly faster than RIFT, due to the binary pattern string construction methods and is steady in all the six test pairs.

Modified RIFT can effectively address the problem of non-linear radiance differences, but it does not achieve good performance when the sensed and reference images have significantly different spatial resolution. Further, the processing time of modified RIFT still needs to be optimized in a future study.

4.3. Qualitative Analysis of the Registration Accuracy

The checkboard-displaying method is adopted to analyze the registration accuracy, qualitatively. Figure 12 displays the stacking image of TIR-visible pairs, with enlarged views of five registration methods. There are obvious misalignments between the TIR and visible image before registration. MI eliminates most of the misalignments and there are still a few stitching errors. HOPC achieves good performance in pair1 and pair2, and fails in pair3. CMM-Net also has unsatisfactory registration results in pair3. The reason for the bad registration accuracy is that the corresponding points are insufficient and imprecise. RIFT acquires good registration results and removes little stitching errors in pair6. The modified RIFT achieves ideal registration accuracy in all test pairs.

4.4. Quantitative Analysis of the Orthorectification Accuracy

The effectiveness of modified RIFT for TIR-visible image registration has been proved in the above discussion. In this section, the geometric accuracy of the SDGSAT-1 TIS image after orthorectification is discussed. Distributed uniformly, 52 GCPs, which are in the image, are selected manually. The root-mean-square error (RMSE), which is defined as follows, is used to evaluate the orthorectification accuracy:

$$\mathrm{RMSE}=\sqrt{\frac{1}{n}{\displaystyle \sum }_{i=1}^n\left[{\left(x_i-\widetilde{x_i}\right)}^2+{\left(y_i-\widetilde{y_i}\right)}^2\right]}$$

(16)

where $\left(x_i,y_i\right)$ are the geographic coordinates in the TIR image, and $\left(\widetilde{x_i},\widetilde{y_i}\right)$ are the geographic coordinates in the reference visible image.

The RMSE in each GCP is calculated and the total RMSE is 1.98 pixels. Figure 13 displays the distribution of GCPs. The GCPs are divided into two types. The GCP plotted in green means that its error is less than 2.5 pixels and the GCP plotted in red means that its error is more than 2.5 pixels. The GCPs, whose errors are more than 2.5 pixels, are distributed in both sides of the images and the mountainous area.

Because the TIR orthoimage is generated by the refinement RPCs, the orthorectification error may result from the original RPCs and the spatial distribution of GCPs. The SDGSAT-1 is currently undergoing in-orbit testing, and there is still internal distortion of the image, which makes the initial RPCs inaccurate. By using modified RIFT to get enough GCPs, and the two-step orthorectification framework, the SDGSAT-1 TIR L4 image production can attain ideal geometric accuracy. With the ongoing work of in-orbit geometric calibration, and the elimination of internal distortion, the geometric accuracy of the SDGSAT-1 TIR image will be further improved by using the method proposed in this paper.

5. Conclusions

In this study, we propose a process flow of the SDGSAT-1 TIS image, from L1A to L4 orthoimage production. Aiming to resolve the problem of unclear details in the TIR image, we employ the homomorphic filtering to enhance image contrast before registration, which can improve the TIR-visible registration performance. Aiming at the problem of the non-linear radiation difference between TIR and visible images, we propose the modified RIFT to registration and GCP extraction. This proposed method has better performance than traditional registration methods. The two-step orthorectification framework is proposed for the SDGSAT-1 TIS orthoimage production. Firstly, the initial RPCs and DEM are adopted for orthorectification, to eliminate the image internal distortion and obtain rough geographic coordinates. Then, the image is divided and registered with the visible reference image to attain enough uniformly distributed GCPs. Finally, the RPCs are refined with the GCPs and the L1A TIR image is orthorectified, again, to generate the L4 orthoimage. The validating GCPs, with uniform spatial distribution, are selected manually, to evaluate the orthorectification accuracy. The accuracy results indicate that the SDGSAT-1 TIS orthoimage is acceptable for the subsequent applications. All of the SDGSAT-1 images in the above experiment are used for the in-orbit testing data. With the ongoing work of in-orbit geometric calibration, the geometric accuracy of SDGSAT-1 TIS images will be further improved.