Geometric Accuracy Evaluation of High-Resolution Satellite Images Based on Xianning Test Field

The evaluation of geometric accuracy of high-resolution satellite images (HRSIs) has been increasingly recognized in recent years. The traditional approach is to verify each satellite individually. It is difficult to directly compare the difference in their accuracy. In order to evaluate geometric accuracy for multiple satellite images based on the same ground control benchmark, a reliable test field in Xianning (China) was utilized for geometric accuracy validation of HRSIs. Our research team has obtained multiple HRSIs in the Xianning test field, such as SPOT-6, Pleaides, ALOS, ZY-3 and TH-1. In addition, ground control points (GCPs) were acquired with GPS by field surveying, which were used to select the significant feature area on the images. We assess the orientation accuracy of the HRSIs with the single image and stereo models. Within this study, the geometrical performance of multiple HRSIs was analyzed in detail, and the results of orientation are shown and discussed. As a result, it is feasible and necessary to establish such a geometric verification field to evaluate the geometric quality of multiple HRSIs.


Introduction
During the last decade, with the improvement of the resolution of HRSIs (Table 1), whose geometric positioning capabilities have also been gradually enhanced, the use of satellite images to produce large-scale topographic maps has become possible. Due to the distinctive feature of high-precision positioning of HRSIs, as well as the stringent requirements of high accuracy and high reliability in mapping, establishing a high stability and long-term continuous-operation scientific testing base for remote sensing and mapping will have significance and practical value. Many research institutions, as well as experts and scholars, have done a lot of investigation and research work for the geometric calibration and validation of sensors. The Modular Optoelectronic Multispectral Scanner (MOMS-2P) was developed by DLR (Deutsches Zentrum für Luft-und Raumfahrt). The DLR updated the calibration data combined with photogrammetric bundle adjustment using an adapted functional model for the reconstruction of the interior orientation. In addition, it also compares the results of geometric laboratory calibration. The calibration field is located in the southern part of Germany and Austria [1,2]. Fraser et al. use different model to process the IKONOS images. The results can yield 3D object-point determination with an accuracy of 0.5 m in plane and 0.7 m in height. The GCPs are collected at road roundabouts or other distinct features conducive to high-precision measurement in both the imagery and on the ground [3]. Tadono et al. describe the updated plans for sensor calibration and product validation of the Panchromatic Remote-sensing Instrument for Stereo Mapping (PRISM), which is to fly on the Advanced Land Observing Satellite (ALOS). They not only evaluate the geometric accuracy of PRISM data, but also validate the derived DEM [4]. The Finnish Geodetic Institute has maintained a permanent test field for geometric, radiometric, and spatial resolution calibration and the testing of high-resolution airborne and satellite imaging systems in Sjökulla since 1994. The experience of nearly 10 years has shown that the use of gravel, combined with appropriate markers as the control objectives of the test field, can effectively eliminate the effects of seasonal and weather changes. It is also durable and able to guarantee the stability and consistency of the test field [5]. Meguro and Fraser evaluated a stereo pair of pansharpened GeoEye-1 Basic images covering the Tsukuba Test Field in Japan, which contains more than 100 precisely surveyed and image-identifiable GCPs. They indicated that the direct georeferencing accuracy is 2 m (CE90, the circular error of above 90% points) in plane and 3 m (LE90, the line error of above 90% points) in height. The use of a few GCPs improved the geopositioning accuracy to around 0.35 m (0.7 pixel) in plane and 0.7 m (1.4 pixel) in height [6]. John Dolloff, et al. use the Metric Information Network (MIN) method to process all 50 WorldView-1 stereo pairs. Statistics based on 101 ICPs (Independent Check Points) show that the positioning result is 0.5 m in plane and 0.3 m in height [7]. G. Agugiaro et al. evaluated the accuracy of GeoEye-1 and WorldView-2 by control and check data of the Trento test field in Italy. Also, 3D information extraction of the images was mentioned. For reference and validation, a DSM (Digital Surface Model) from airborne LiDAR acquisition is used as a comparison [8]. H. Topan and D. Maktav validated that different variations of point distribution and EOP configuration were preferred, achieving georeferencing accuracies of ±1 m and~±5 m at control and check points, respectively [9]. Wang et al. validated that ZY-3 can be used for the generation of cartographic maps at the 1:50,000 scale and for revision and updates of 1:25,000 scale maps [10][11][12][13][14]. By detecting and eliminating various kinds of geometric processing error, including equipment installation error, attitude and orbit measurement error, camera distortion, time synchronization errors and other errors, Li et al. found that the geometric orientation accuracy of Chinese satellite images could be improved to be better than 1.5 pixel, which is higher than the designed accuracy [15]. Tian et al. showed that more accurate and reliable assessment results can be obtained by choosing the appropriate evaluation method of geometric positioning accuracy [16].
Worldwide, scholars have done a lot of geometric accuracy verification work with multiple types of HRSIs. In previous research processes, the evaluation area and evaluation method of each satellite are different. The geometric positioning accuracy of each satellite could only be compared by related reports and papers. A unified test field is hoped to be established and a unified evaluation method is used to evaluate the geometric positioning accuracy of different satellites. The Xianning test field can meet this demand. Data collection, processing, and evaluation processes will become standardized with the Xianning test field, and the accuracy of the evaluation results can be compared more simply and intuitively.
Moreover, the designers of the test field will generally consider satellite geometry calibration and validation work in the same field. The internal and external orientation parameters of the satellite will be updated by the calibration. However, it requires a time lag after calibration in order to ensure the reliability of the accuracy. Therefore, a special validation test field is necessary only for accuracy evaluation using the ICPs (Independent Check Points) in the test field. In addition, this validation test field is different from the calibration test field. The main purpose of the validation test field is to verify the geopositioning accuracy and object recognition capabilities of the standard HRSI, and also to provide a reference for the direct application of the HRSI. The Xianning test field is such a test field for accuracy validation of HRSI or other remote sensing data.

The Test Field
The test area is located in Xianning, a city of about 2,880,000 inhabitants in the south of Hubei province, central of China. The test field is situated in plain and hilly topography and the elevation ranges from 20 m to 400 m. This topographic feature is representative of flat and hilly land in China. The test field varies from urban areas with residential, industrial and commercial buildings at different sizes and heights, to agricultural or forested areas, and steep rocky surfaces, therefore offering a heterogeneous landscape in terms of geometrical complexity, land use and cover. Also, it needs convenient transportation and to be away from crowded areas in order to ensure that no deformation of the surface features will take place over time.
The area with the GCPs in the test field should contain at least two standard scenes to be able to make the most of the width of whole satellite images within the scope of the test field. Therefore, it is believed that the width of test field should be about 120 km (based on the value of double the current maximum width of the HRSIs in Table 1). A length of the test field of about 100 km along the track direction would be most appropriate; also double the standard length of one HRSI. Currently, the Xianning test field can meet the requirements of the width and length of most mainstream HRSIs. Thumbnail of multiple HRSIs coverage is shown in Figure 1. are different. The geometric positioning accuracy of each satellite could only be compared by related reports and papers. A unified test field is hoped to be established and a unified evaluation method is used to evaluate the geometric positioning accuracy of different satellites. The Xianning test field can meet this demand. Data collection, processing, and evaluation processes will become standardized with the Xianning test field, and the accuracy of the evaluation results can be compared more simply and intuitively. Moreover, the designers of the test field will generally consider satellite geometry calibration and validation work in the same field. The internal and external orientation parameters of the satellite will be updated by the calibration. However, it requires a time lag after calibration in order to ensure the reliability of the accuracy. Therefore, a special validation test field is necessary only for accuracy evaluation using the ICPs (Independent Check Points) in the test field. In addition, this validation test field is different from the calibration test field. The main purpose of the validation test field is to verify the geopositioning accuracy and object recognition capabilities of the standard HRSI, and also to provide a reference for the direct application of the HRSI. The Xianning test field is such a test field for accuracy validation of HRSI or other remote sensing data.

The Test Field
The test area is located in Xianning, a city of about 2,880,000 inhabitants in the south of Hubei province, central of China. The test field is situated in plain and hilly topography and the elevation ranges from 20 m to 400 m. This topographic feature is representative of flat and hilly land in China. The test field varies from urban areas with residential, industrial and commercial buildings at different sizes and heights, to agricultural or forested areas, and steep rocky surfaces, therefore offering a heterogeneous landscape in terms of geometrical complexity, land use and cover. Also, it needs convenient transportation and to be away from crowded areas in order to ensure that no deformation of the surface features will take place over time.
The area with the GCPs in the test field should contain at least two standard scenes to be able to make the most of the width of whole satellite images within the scope of the test field. Therefore, it is believed that the width of test field should be about 120 km (based on the value of double the current maximum width of the HRSIs in Table 1). A length of the test field of about 100 km along the track direction would be most appropriate; also double the standard length of one HRSI. Currently, the Xianning test field can meet the requirements of the width and length of most mainstream HRSIs. Thumbnail of multiple HRSIs coverage is shown in Figure 1.

The GCPs
Precision ground control data is the guarantee of geometric accuracy validation of HRSIs. Our group has done related work about GCPs laid in the Xianning region. A total of 118 GCPs are located in the test field as shown in Figure 2. Natural and artificial object feature points are chosen as GCPs by a static GPS field survey. The control points can be evenly distributed in each image, and the accuracy is about 10 cm, which is able to fully meet the needs of control and check points for all types of HRSIs.

The GCPs
Precision ground control data is the guarantee of geometric accuracy validation of HRSIs. Our group has done related work about GCPs laid in the Xianning region. A total of 118 GCPs are located in the test field as shown in Figure 2. Natural and artificial object feature points are chosen as GCPs by a static GPS field survey. The control points can be evenly distributed in each image, and the accuracy is about 10 cm, which is able to fully meet the needs of control and check points for all types of HRSIs.

Multiple HRSIs Data
Our team acquired 5 kinds of HRSIs within the range of the Xianning test filed. The HRSI data are summarized in Table 2, and include: The processing level is primary product. Primary product is the processing level closest to the natural image acquired by the sensor. This product restores perfect collections: the sensor is placed in rectilinear geometry, and the image is clear of all radiometric distortion. RPCs (Rational Polynomial Coefficients) and the sensor model are provided with the product. The data is a standard image product which includes nadir-forward-backward panchromatic images.  SPOT-6: The processing level is primary product. The image is corrected for radiometric and sensor distortions, using internal calibration parameters, ephemeris and attitude measurements.
RPCs and the sensor model are provided with the product. The data is a standard image product which includes nadir-forward-backward panchromatic images.  ALOS PRISM: The processing level is 1B1 product. On the basis of Level 1A, the data with radiometric correction and added absolute calibration coefficient. RPCs and the sensor model are provided with the product. The data is a standard image product which includes nadirforward-backward panchromatic images.  ZY-3: The processing level is Sensor Corrected, i.e., the images are radiometrically and sensor corrected, but not projected to a plane using a map projection or datum, thus keeping the original acquisition geometry. The images were provided with RPCs. The data is a long strip image product which includes nadir-forward-backward panchromatic images [17].  TH-1: the processing level is 1B; for each image, the RPCs were provided. The images are radiometrically and sensor corrected. The data is a standard image product which includes nadir-forward-backward panchromatic images. Other details about the characteristics of various satellite sensors are listed in Table 2.

Multiple HRSIs Data
Our team acquired 5 kinds of HRSIs within the range of the Xianning test filed. The HRSI data are summarized in Table 2, and include:

•
Pleiades: The processing level is primary product. Primary product is the processing level closest to the natural image acquired by the sensor. This product restores perfect collections: the sensor is placed in rectilinear geometry, and the image is clear of all radiometric distortion. RPCs (Rational Polynomial Coefficients) and the sensor model are provided with the product. The data is a standard image product which includes nadir-forward-backward panchromatic images. • SPOT-6: The processing level is primary product. The image is corrected for radiometric and sensor distortions, using internal calibration parameters, ephemeris and attitude measurements.
RPCs and the sensor model are provided with the product. The data is a standard image product which includes nadir-forward-backward panchromatic images. • ALOS PRISM: The processing level is 1B1 product. On the basis of Level 1A, the data with radiometric correction and added absolute calibration coefficient. RPCs and the sensor model are provided with the product. The data is a standard image product which includes nadir-forward-backward panchromatic images. • ZY-3: The processing level is Sensor Corrected, i.e., the images are radiometrically and sensor corrected, but not projected to a plane using a map projection or datum, thus keeping the original acquisition geometry. The images were provided with RPCs. The data is a long strip image product which includes nadir-forward-backward panchromatic images [17]. • TH-1: the processing level is 1B; for each image, the RPCs were provided. The images are radiometrically and sensor corrected. The data is a standard image product which includes nadir-forward-backward panchromatic images. Other details about the characteristics of various satellite sensors are listed in Table 2.

General Geometric Processing Model of HRSIs
As seen in Section 2, all HRSIs are provided with RFM (Rational Function Model). Toutin and Teo's study has shown that the result of satellite image orientation based on the RFM is almost as accurate as that based on the rigorous geometric model [18,19]. Additionally, the RFM has a simple form and leads to fast computations, so the orientation selects the RFM as the geometric model. The RFM describes the relations between the image point coordinates (r n , c n ) and the ground point coordinates (X n , Y n , Z n ), which have the following general form [20]: where (r n , c n ) are measured line and sample coordinates of the nth image point, corresponding to the ground point with the object space coordinates (X n , Y n , Z n ), which are the variables of a polynomial p i (i = 1, 2, 3, 4), whose degree should not exceed three. For example, the form of the polynomial p i is where a ij (i = 1, 2, 3, 4, j = 0, 1, · · · , 19) are rational polynomial coefficients [20]. Furthermore, bias compensation is needed. Previous studies have shown that the affine model combined with RFM can eliminate the systematic errors in the image points, which improves the geometry processing accuracy [11]. Therefore, we modify the relationship between the image coordinates (x, y) and the coordinates (X, Y, Z) according to Formula (3) [21].
The affine transformation parameters (a 0 , a 1 , a 2 , b 0 , b 1 , b 2 ) are set as orientation parameters. It can be solved with a small number of GCPs.

Validation of Geometric Accuracy for HRSIs
The test is expected to demonstrate to what extent direct georeferencing and sensor orientation are accurate and efficient methods for the determination of the exterior orientation parameters for topographic mapping. Currently, such assessments are performed through the validation technique known as the Hold-Out Validation (HOV) method [22]. It is also known as test sample estimation. According to this, the data set (known ground points) is partitioned into two subsets: the first one is used in the orientation model (GCPs-Ground Control Points) and the second one is used to validate the model itself (ICPs-Independent Check Points).

Orientation Accuracy with Single Image
The image orientation determines the relation between the object and the image coordinates, which is dependent on the image product and the imaging mode. So at least for reliability, GCPs are required. The image orientation can be based on a geometric reconstruction of the imaging geometry, depending upon the available information. The direct sensor orientation may be available, too, as a sensor-oriented RPC. Like the geometric reconstruction, this can be improved by GCPs, named and also bias corrected [21].
The orientation accuracy was analyzed and verified based different schemes of laid GCPs, as follows in Table 3: It can be seen that Pleaides and SPOT6 performed with the highest accuracy without GCPs, almost reaching the 3 pixel level, from Table 3. The results of ZY-3 and TH-1 are almost the same, reaching about 6 pixels. When 4 GCPs were laid in the four corners, ZY-3, SPOT6 and Pleaides all reached the 1.5 pixel level or better. However, residuals of some points in the TH-1 image were still large after orientation with GCPs. The interior geometric accuracy of TH-1 is poor, as can be seen from the residuals distributions in Figure 3. More GCPs were added, but no more obvious changes occurred. Therefore, the scheme with 4 GCPs located in each corner is recommended for HRSI orientation.  It can be seen that Pleaides and SPOT6 performed with the highest accuracy without GCPs, almost reaching the 3 pixel level, from Table 3. The results of ZY-3 and TH-1 are almost the same, reaching about 6 pixels. When 4 GCPs were laid in the four corners, ZY-3, SPOT6 and Pleaides all reached the 1.5 pixel level or better. However, residuals of some points in the TH-1 image were still large after orientation with GCPs. The interior geometric accuracy of TH-1 is poor, as can be seen from the residuals distributions in Figure 3. More GCPs were added, but no more obvious changes occurred. Therefore, the scheme with 4 GCPs located in each corner is recommended for HRSI orientation. Residual distribution figures are shown below in Figures 4-8:    It can be seen that Pleaides and SPOT6 performed with the highest accuracy without GCPs, almost reaching the 3 pixel level, from Table 3. The results of ZY-3 and TH-1 are almost the same, reaching about 6 pixels. When 4 GCPs were laid in the four corners, ZY-3, SPOT6 and Pleaides all reached the 1.5 pixel level or better. However, residuals of some points in the TH-1 image were still large after orientation with GCPs. The interior geometric accuracy of TH-1 is poor, as can be seen from the residuals distributions in Figure 3. More GCPs were added, but no more obvious changes occurred. Therefore, the scheme with 4 GCPs located in each corner is recommended for HRSI orientation. Residual distribution figures are shown below in Figures 4-8:

Orientation Accuracy of Block Adjustment with Tri-Stereo Images
All these HRSIs are able to constitute a three linear array stereo model with nadir-forward-backward images. In consideration of the multi-covered image data with redundant observations information,

Orientation Accuracy of Block Adjustment with Tri-Stereo Images
All these HRSIs are able to constitute a three linear array stereo model with nadir-forward-backward images. In consideration of the multi-covered image data with redundant observations information,

Orientation Accuracy of Block Adjustment with Tri-Stereo Images
All these HRSIs are able to constitute a three linear array stereo model with nadir-forward-backward images. In consideration of the multi-covered image data with redundant observations information,

Orientation Accuracy of Block Adjustment with Tri-Stereo Images
All these HRSIs are able to constitute a three linear array stereo model with nadir-forward-backward images. In consideration of the multi-covered image data with redundant observations information,

Orientation Accuracy of Block Adjustment with Tri-Stereo Images
All these HRSIs are able to constitute a three linear array stereo model with nadir-forward-backward images. In consideration of the multi-covered image data with redundant observations information, block adjustment was carried out with and without GCPs to evaluate the orientation accuracy. Conclusive results are as follows: From Table 4 it can be seen that in the situation of no GCP, Pleaides reached an accuracy of 0.860 m in plane and 2.654 m in height, which is really unbelievable, although the 0.5 m GSD (Ground Sampling Distance) of Pleaides is the highest among the HRSIs. SPOT6 reached an accuracy of 5.336 m in plane and 4.595 m in height, and also has a superior performance in geometric accuracy without GCP. The block adjustment accuracy without GCP of Pleaides and SPOT6 meet the requirements for 1:50,000 Topographic maps. However, ALOS, ZY-3 and TH-1 cannot reach that level. From Table 4 it can also be seen that in the situation with GCPs, when four GCPs were laid in the four corners, the block adjustment accuracy of ZY-3 reaches 2.653 m in plane and 1.858 m in height, and ALOS reaches 2.900 m in plane and 1.363 m in height. Compared to the accuracy without GCP, this accuracy is improved a lot, reaching the level of SPOT6, with the accuracy in height being even better. Pleaides still has the best performance, while TH-1 has the worst performance, reaching 17.152 m in plane and 7.467 m in height, although its resolution of 5 m is the lowest. Under the condition of setting the GCPs in the four corners, without consideration of artificial pricking points, the accuracy almost reaches the best level for single image orientation and for block adjustment. The accuracy undergoes no more changes, even when more GCPs are added. Therefore, four GCPs laid in the corners is a good layout scheme, which is recommended.
From another point of view, if a satellite image cannot achieve an accuracy of 1-2 pixels with four GCPs laid, it can be shown that the internal geometric distortion has not been eliminated before the generation of standard image products. For satellite images without internal geometric distortion, the four GCPs laid can eliminate most of the errors and achieve high-precision positioning.

Conclusions
Different from the traditional methods of verification, the paper has embarked on a unified test investigating sensor orientation and describes the processing carried out on ZY-3, TH-1, ALOS, SPOT6 and Pleaides in a geometric accuracy test field, instead of verifying them separately. All the results are compared under conditions both without and with GCPs, and whether in orientation with single image or in block adjustment. The performance of Pleaides is the best. SPOT6, ALOS and ZY-3 are almost comparable, although the ground resolution of SPOT6 is slightly higher than that of ZY-3. The performance of TH-1 is a little worse. A layout scheme of four GCPs laid in the corners is recommended, and can be used for geometric precision processing and evaluation of HRSIs.
Also, the test field set up in Xianning is presented with the aim of investigating spaceborne optical imagery. The reason for choosing the Xianing area as the test field is explained in detail, and the function of the Xianning test field is also illustrated for the geometric accuracy validation of HRSIs. The test field will undoubtedly be important both for development, analysis, and simulation of platforms and sensors in the future. In addition, verification work of the HRSI data obtained will continue to be carried out.