A New Method for Remote Sensing Satellite Observation Effectiveness Evaluation

Abstract

The number of remote sensing satellites has increased rapidly in parallel with the advancement of space technology and the rising demand in the space industry. Consequently, the observation effectiveness evaluation of remote sensing satellites has received extensive attention. As the core content of the effectiveness evaluation, index systems are usually established and screened using qualitative or quantitative methods. They can hardly satisfy the construction principles such as completeness and independence simultaneously. To address this issue, we propose a new method for remote sensing satellite observation effectiveness evaluation that considers various principles. Firstly, a three-layer evaluation index system structure is constructed. The principle of completeness, hierarchy, and measurability of the index system is ensured by decomposition, clustering, and preliminary screening. Secondly, the principal component contribution rate is obtained through principal component analysis. Finally, we introduce a comprehensive scoring method (ICCLR) based on the combination of independence coefficient and principal component comprehensive loss rate. It realizes the screening of an index system from the index set containing correlation relationships. The validity and optimality of the proposed method are verified through experiments and analysis of three typical tasks.

1. Introduction

Remote sensing satellites acquire information about land, ocean, meteorology, etc., by using various payloads [1,2]. They can perform global earth observations without regional restrictions, and thus play an important role in the remote sensing field [3]. Due to the recent advancement in satellite development, the number of remote sensing satellites has drastically risen, growing from less than 400 satellites in 2012 to more than 1000 satellites in 2022 [4,5]. Remote sensing satellites distributed in different orbital positions usually use attitude maneuvers to provide high-frequency and high-resolution observation services, including point target, area target, and moving target observation tasks. However, owing to the complexity of satellites’ composition and structure, as well as the varying task environments, the full utilization of valuable satellite resources is challenging [6]. Hence, it is necessary to comprehensively evaluate the observation effectiveness of remote sensing satellites, so as to ensure the optimality of the decision-making process such as tasks allocation and attitude maneuver [7,8].

The study of effectiveness evaluation is mainly divided into two parts: the construction of the index system and the research of evaluation methodologies [9]. The index system is a whole composed of several related indices, which can completely reflect the characteristics of the evaluation object from all aspects [10]. As the main content of the evaluation study, the reasonability of the index system will have a direct impact on the effectiveness of the evaluation [11]. Additionally, the complex diversity of satellite systems and tasks introduces more challenges to the applicability of the index system. Therefore, it is crucial to determine whether the index system can comprehensively and effectively reflect the system’s ability to accomplish various tasks [12].

The construction of an index system typically contains the following basic principles [13,14]: (1) completeness: the index system should cover all the attributes of the evaluation object as much as possible. (2) Hierarchy: the index system should have a clear hierarchy structure where the relationships between indices in each layer are definite. (3) Measurability: each index can be quantitatively calculated to obtain a numerical value that intuitively reflects the system’s capability. (4) Independence: each index should be as uncorrelated as possible to avoid redundant information. (5) Simplicity: the index system should not include too many indices. It should only use the least number of indices to cover the most attributes. In respect of the completeness problem, an index system with insufficient completeness will lead to one-sided cognition, causing defects in design, manufacture, and application [14]. Furthermore, in terms of independence and simplicity, several redundant or relational indices might exacerbate the difficulties in determining the weight value [15,16,17,18]. Therefore, the screening of indices is a necessary part of the index system construction process [19].

Current research in the construction of index systems is categorized into qualitative and quantitative methods. The qualitative methods mainly include the expert experience (Delphi) method and the analytic hierarchy process (AHP) method. Regarding the Delphi method, the index systems are usually obtained by experts solely based on their experience and analysis of the research object. The index systems can well reflect all aspects of the system capabilities but inevitably include redundant indices. Zheng et al. [10] established an evaluation system including revisit capability, observation interval, observation frequency, observation duration, and other indices from the perspective of three target types: point target, area target, and moving target. Barsi et al. [20] established a remote sensing data quality index system including resolution, accuracy, completeness, redundancy, readability, and accessibility. Liu et al. [21] developed an index system with a total of eighteen indices in four categories including coverage efficiency, mission planning efficiency, communication efficiency, and resource scheduling efficiency. To evaluate the feasibility and effectiveness of nanosatellites in military operations, Tang et al. [22] developed an index system with fifteen indices divided into four categories, including space segment capability, ground segment capability, survivability, and command and control capability. Elhady, A.M. [23] constructed an index system that includes instantaneous availability, mission availability, steady state of availability, achieved availability, and intrinsic availability. Additionally, Li et al. [24] established an index system consisting of six indices of temporal resolution and spatial resolution for the evaluation of remote sensing satellite coverage effectiveness. Regarding the satellite observation and data-downlink scheduling problem, Zhang et al. [8] suggested an Availability–Capacity–Profitability (ACP) evaluation framework. Regarding the AHP method, the index systems are usually scored by experts for index screening. Although this method can effectively reduce the number of indices, the evaluation of the loss of completeness and independence is arduous. Li et al. [25] established an index system for the coordination degree of meteorological satellite stakeholder relationship network and used the AHP method to determine the index weights and screen these indices.

Regarding the quantitative research aspect, the index system is generally obtained through numerical statistical analysis and index screening. The quantitative methods mainly include correlation analysis, information entropy theory, and principal component analysis (PCA) method. Correlation analysis is an effective method to determine the index redundancy relationship and reduce the number of factors. Li et al. [26] established an index system for the observation effectiveness of satellites on moving targets with the use of correlation analysis to reduce redundant indices. Aiming at the evaluation problem of the space information network demonstration platform, Liu et al. [14] carried out an index screening method by calculating the correlation coefficient and comprehensive weight to ensure the independence of the index system. Generally, correlation analysis can assure the independence and simplicity of the index system but cannot evaluate the loss of completeness. Regarding the PCA method, it is a common approach to address the simplicity and independence problem. However, the resulting orthogonal principal components lack clear physical meaning, thus, fail to meet the intuition requirements of the measurability principle. To address the redundancy problem of the primary index set in the armored vehicle (prognostics and health management) PHM system performance index system, Zhai et al. [19] used the PCA method to select the indices. Finally, the index screening method based on information entropy theory usually selects the index according to the amount of information. A representative method is the entropy weight method (EWM). Huang et al. [27] combined the AHP and EWM methods to screen out components that are closely associated with the efficacy of drugs. Based on the information entropy theory, Wang et al. [10] defined the direction loss rate to measure the evaluation ability loss of the index screening results. Normally, the EWM method can evaluate and ensure the completeness of the index system.

Most of the studies mentioned above fail to consider all principles concurrently. In particular, the principles of completeness, independence, and simplicity of the index system contradict themselves and are thereby unable to be considered simultaneously. Hence, it is practically challenging to use the least possible number of independent indices to reflect the system’s capability in every aspect.

In this paper, in an attempt to address the effectiveness evaluation problem of remote sensing satellites, an index system construction method is proposed. This method fully considers the five principles: completeness, hierarchy, measurability, independence, and simplicity. Firstly, according to the requirement analysis of remote sensing satellites, a three-layer structure is suggested. This structure comprises a system layer, a task layer, and an index layer. Secondly, an initial index system set that is complete, hierarchical, and measurable is constructed through index decomposition, clustering, and preliminary screening. Finally, index sets in every task layer are screened quantitatively. Based on PCA, the ICCLR approach that extracts the index system is then proposed by introducing the calculation methods of independence coefficients and the comprehensive loss rate of the contribution rate. From index sets that are redundant and correlated, this approach retrieves index systems that fulfill the principle of completeness, independence, and simplicity. Through the simulation experiments and quantitative analysis of three typical mission scenes of remote sensing satellites, a complete evaluation index system is constructed. By comparing our proposed method with the conventional research, the effectiveness of the proposed method is verified.

The remaining parts of this paper are organized as follows. In Section 2, the method for remote sensing satellite observation effectiveness evaluation is described. Section 3 begins with the formation of three simulation scenes, followed by the index screening method using quantitative computation to generate an index system that considers all of the principles aforementioned. In Section 4, a discussion of the experimental results is provided. Lastly, Section 5 concludes this paper.

2. Materials and Methods

Remote sensing satellite systems are typically composed of optical satellites, synthetic aperture radar (SAR) satellites, and satellites carrying other payloads. They collaborate to carry out observation missions of worldwide targets. Targets are normally classified as point targets, area targets, and moving targets. The observation mission scene of remote sensing satellites is shown in Figure 1.

The evaluation index system for remote sensing satellites is constructed from the mission perspective. As shown in Figure 2, the method mainly consists of four steps: (1) index system hierarchy construction, (2) index decomposition and clustering, (3) index system preliminary screening of measurability and independence, and (4) index system quantitative screening considering completeness, independence, and simplicity.

2.1. Index System Hierarchy Construction

The first layer of the index system is defined as the system layer which represents the comprehensive observation capability. Comprehensive observation capability is the comprehensive value of the ability of remote sensing satellites in accomplishing different observation tasks. This value can be obtained by weighting the underlying indices.

Next, the remote sensing satellite observation task is decomposed according to three main targets, which are the point target, area target, and moving target. As the target characteristics and observation requirements are noticeably distinct, the emphasis on effectiveness evaluation for each target is different. Therefore, by referring to the three different tasks, the task layer is defined as the second layer of the index system.

Each task layer contains numerous dissimilar effectiveness indices. Although some of them are hierarchically related, the number of layers in the index system should be limited. Hence, the index system only retains a three-layer hierarchy structure. The third construction layer is the index layer, which is the smallest unit in the index system.

The three-layer structure of the index system is shown in Figure 3.

2.2. Index Decomposition and Clustering

The effectiveness indices are decomposed based on temporal resolution, spatial resolution, spectral resolution, and radiometric resolution. The index decomposition result is presented in the first three columns in Figure 4 [10,20,21,22,23,26]. The temporal resolution contains various indices, including revisit time, observation time interval, single observation ability, total coverage time, task response-ability, and continuous observation ability. The spatial resolution includes ground sampling distance (GSD), total observation number, coverage percentage, and capability of discovery, identification, and tracking. Spectral resolution consists of spectral range, band number, and band interval.

The characteristics of three different tasks and the applicable index set are then analyzed. The clustering results are summarized in the last three columns of Figure 4, where Task 1, Task 2, and Task 3 represent the observation tasks of the point target, area target, and moving target, respectively. The detailed analysis and clustering process are as follows.

(1) Point target observation effectiveness index.

Point targets generally refer to small, fixed targets on the ground, such as airports, ports, buildings, etc. The dimensions of the point target are much smaller than the field of view of the remote sensing satellite; thus, only a single observation is needed to fully cover the point target. The requirements of observation effectiveness for point targets are mainly reflected in the coverage, timeliness, and imaging capabilities. Consequently, the main indices used to evaluate the effectiveness of point target observation are: revisit time, observation time interval, coverage percentage, coverage time rate, GSD, number of observations, spectral resolution, and radiometric resolution.

(2) Area target observation effectiveness index.

Area targets are generally of a certain scale, such as rivers, forests, and countries’ frontiers. As the dimensions of the target are relatively large, a single observation is inadequate to cover the whole target. Several satellites are therefore required to cooperate so that the target can be completely covered through multiple observations. The observation effectiveness of the area target is similar to that of the point target, but the numerical patterns for some of the same indices are different. Take the observation time interval and single observation time as examples. The larger the target dimensions, the longer the observation time. In addition, the full coverage ability for the area target is added to the evaluation index system, particularly the maximum coverage interval and average coverage interval.

(3) Moving target observation effectiveness index.

Moving targets include ships, vehicles, and other moving objects. Due to their small dimensions and motion uncertainty, close cooperation among multiple satellites are necessary to observe the moving targets. The typical moving target observation process begins with the use of wide-swath satellites to scan and search for the target. Next, high-resolution satellites are used to recognize and identify the moving targets. Subsequently, the remote sensing satellites will track the target continuously. The observation effectiveness of moving targets is mainly indicated in the success rate and timelines of different tasks. Hence, the index clustering results include observation time intervals, as well as the probability and response time of the three processes. These processes are search and discovery, identification and confirmation, and also continuous tracking.

2.3. Index System Preliminary Screening

The preliminary screening principle of the index system includes measurability and independence. The principle of measurability requires that the indices can be quantitatively calculated so that the indices can intuitively and objectively reflect the system’s capabilities. Independence requires the indices to be highly uncorrelated so that the index system’s hierarchy structure will be unaffected. Note that the indices of spectral resolution and radiometric resolution only describe the static capabilities of the payload. As they do not vary with time and tasks, they are mutually independent of other indices related to temporal resolution and spatial resolution. In the subsequent index screening, indices of spectral resolution and radiometric resolution are excluded. The preliminary screening process is as follows:

(1) Measurability screening, eliminate indices that cannot effectively measure the system's capabilities.

The overall tracking capability index is defined as the capability of continuously tracking the entire process of the target from appearing to disappearing in a specific area. As the satellite system is susceptible to weather, clouds, and other factors, it is generally impossible to track the whole process of the target. This index can hardly reflect the system’s capability since it has only a value of either 0 or 1. In contrast, tracking time percentage is the index that can effectively reveal the system’s capability for continuous tracking of the moving target. To comply with the measurability principle, the overall tracking capability index is eliminated from the index system.

(2) Independence screening, remove highly correlated and linearly dependent indices through correlation analysis.

There is a correlation between the total observation duration index and the coverage time rate index as the total observation duration is equal to the product of total time, total target number, and coverage time rate. The total observation duration index is an absolute time with no clear meaning as it is affected by the changes in the total time and the total target number. Instead, the coverage time rate index is a relative value that is more general in use. Hence, the total observation duration index is removed.

The average single observation time, average revisit time, and average observation interval are three correlated indices. The average single observation is redundant because it is obtained by subtracting the average observation time from the average revisit time. On the other hand, the revisit time and observation interval have three conventional indices, which are the maximum, minimum, and average values. Thus, the average single observation is removed to decrease the correlation among indices.

After screening the indices with the consideration of the principle of measurability and independence, the preliminary screening result of the index system are presented in Figure 5.

2.4. Index System Quantitative Screening (ICCLR)

An index system is gained after the preliminary screening. The completeness, hierarchy, and measurability of the index system are ensured, but the indices are still unavoidably redundant and correlated, which leads to inadequate independence and simplicity. This section presents a comprehensive scoring ICCLR method using the principal component analysis combined with the independence coefficient and the comprehensive loss rate of the contribution rate. After quantitative analysis and calculation, the index sets under each task layer branch are screened to obtain the optimal index system. The quantitative screening process of the index system is presented in Figure 6.

Step 1: Sample collection.

Given the preliminarily screened index sets $S_{ini}$ in the specific task layer, which includes $N$ effectiveness indices.

$$S_{ini}=\left[Inde{x}_1,\dots Inde{x}_i,\dots Inde{x}_N\right],1\le i\le N.$$

(1)

Analyze the systems and task parameters that have a significant impact on the effectiveness indices in the task scenes. Then, the influencing parameters are randomly scattered to generate $M$ simulation conditions. The effectiveness indices of each simulation condition are simulated and calculated. After Z-score normalization, a sample matrix $M_{ini}\in {ℝ}^{M\text{×}N}$ is generated for index screening.

$$M_{ini}=\left[\begin{array}{cccc}{I}_{ini}^{1,1}& I_{ini}^{1,2}& \cdots & I_{ini}^{1,N}\\ I_{ini}^{2,1}& I_{ini}^{2,2}& \cdots & I_{ini}^{2,N}\\ ⋮& ⋮& \ddots & ⋮\\ I_{ini}^{M,1}& I_{ini}^{M,2}& \cdots & I_{ini}^{M,N}\end{array}\right],$$

(2)

where $I_{ini}^{i,j}$ is the calculated value of $j$th index in $i$th sample, note that $1\le i\le M$, $1\le j\le N$.

Step 2: Principal component analysis.

Perform PCA on Matrix $M_{ini}$ to obtain two sets, which are the principal component set $S_{PCA}$ and the contribution rate of the principal component set $C_{PCA}$ [19].

$$S_{PCA}[P{C}_1,\dots P{C}_i,\dots P{C}_N],1\le i\le N.$$

(3)

$$C_{PCA}[C_{P{C}_1},\dots C_{P{C}_i},\dots C_{P{C}_N}],1\le i\le N.$$

(4)

According to a specified requirement of cumulative contribution rate $\alpha$ (normally set to 90–95%), the number of top $N_{\alpha }$ principal components is obtained, satisfying

$${\displaystyle \sum _{i=1}^{N_{\alpha }-1}{C}_{P{C}_i}<\alpha }\mathrm{and}{\displaystyle \sum _{i=1}^{N_{\alpha }}{C}_{P{C}_i}\ge \alpha }.$$

(5)

Meanwhile, the transformation matrix $L_{PI}\in {\mathbb{R}}^{N\times N}$ of the initial index set $S_{ini}$ and principal component set $S_{PCA}$ can also be obtained, $L_{PI}=L_{IP}{}^{\prime }$, $L_{IP}={[{\omega }_1,\dots {\omega }_i,\dots {\omega }_N]}^T$, and ${\omega }_i$ is the feature vector corresponding to $i$th principal component. In the matrix $L_{PI}$, each row is a vector of the initial index in the principal component coordinate system. The above relationship can be expressed as

$$M_{PCA}\cdot L_{PI}=M_{ini}.$$

(6)

Step 3: Traverse to build sub-index sets.

Select $p$ indices from the $N$ indices included in the index set $S_{ini}$, then form ${\mathrm{C}}_N^p$ sub-index sets $S_n$, where $1\le n\le {\mathrm{C}}_N^p$ and

$$S_n=\left[Inde{x}_1,\dots Inde{x}_i,\dots Inde{x}_p\right],1\le i\le p.$$

(7)

Step 4: Calculate the independence coefficient $k_n$ for each sub-index set.

$p$ indices in $S_n$ are projected to the top $p$ principal components in $S_{PCA}$, where the projection matrix is $L_n\in {ℝ}^{p\times p}$. Since only the top $p$ principal components are extracted, the coordinates of $N-p$ principal components can be neglected, and thus the coefficients of $N-p$ rows are deleted from $L_{PI}$. Next, according to the numbering of $p$ indices in $S_n$, select the corresponding rows to obtain $L_n$.

Calculate the independence coefficient $k_n$ of $S_n$, which is the reciprocal condition number of $L_n$.

$$k_n=\frac{1}{\mathrm{cond}(L_n)}=\frac{1}{\Vert L_n^{-1}\Vert \cdot \Vert L_n\Vert }.$$

(8)

The condition number is one of the important indicators of the ill-conditioned matrix. If the condition number is around 1, the matrix is well-conditioned, meaning that the variables are all mutually independent. Conversely, the higher the condition number, the more ill-conditioned the matrix is. Aiming at the effectiveness index screening problem, the reciprocal condition number is defined as the independence coefficient $k_n$. When $k_n$ is closer to 1, the independence of the index system is higher.

Step 5: Calculate the comprehensive loss rate of the contribution rate $C{L}_n$ for each sub-index set.

$C{L}_n$ is defined as the weighted value of the projected distance difference of all the samples on the $N$ principal components, under two different index systems $S_{\mathit{ini}}$ and $S_n$, where the weighting coefficient value is the contribution rate of the principal component. $C{L}_n\in [0,1]$, when the value is lower, the information loss of $S_n$ is lesser, indicating that the index system is more complete. The calculation process is shown below.

Under the complete index system $S_{ini}$, all samples are projected to the principal component coordinate system projection matrix $M_{PCA}\in {ℝ}^{M\text{×}N}$

$$M_{PCA}=M_{ini}\cdot L_{IP}=M_{ini}\cdot L_{PI}^{-1}=\left[\begin{array}{cccc}{I}_{PCA}^{1,1}& I_{PCA}^{1,2}& \cdots & I_{PCA}^{1,N}\\ I_{PCA}^{2,1}& I_{PCA}^{2,2}& \cdots & I_{PCA}^{2,N}\\ ⋮& ⋮& \ddots & ⋮\\ I_{PCA}^{M,1}& I_{PCA}^{M,2}& \cdots & I_{PCA}^{M,N}\end{array}\right].$$

(9)

Under the index system $S_n$, the evaluation result of samples is $M_{ini,n}\in {ℝ}^{M\text{×}p}$, where the transformation matrix is $L_{IP,n}$. The evaluation result is projected to the principal component coordinate system projection matrix $M_{PCA,n}\in {ℝ}^{M\text{×}N}$

$$M_{PCA,n}=M_{ini}\cdot L_{IP,n}=\left[\begin{array}{cccc}{I}_{PCA,n}^{1,1}& I_{PCA,n}^{1,2}& \cdots & I_{PCA,n}^{1,N}\\ I_{PCA,n}^{2,1}& I_{PCA,n}^{2,2}& \cdots & I_{PCA,n}^{2,N}\\ ⋮& ⋮& \ddots & ⋮\\ I_{PCA,n}^{M,1}& I_{PCA,n}^{M,2}& \cdots & I_{PCA,n}^{M,N}\end{array}\right].$$

(10)

$C{L}_n$ is computed by

$$C{L}_n={\displaystyle \sum _{i=1}^N\left(C_{P{C}_i}\cdot \frac{{\displaystyle \sum _{j=1}^M\left|I_{PCA}^{j,i}-I_{PCA,n}^{j.i}\right|}}{M\cdot \left(\underset{j=1}{\overset{M}{\max }}{I}_{PCA}^{j,i}-\underset{j=1}{\overset{M}{\min }}{I}_{PCA}^{j,i}\right)}\right)}.$$

(11)

Step 6: Calculate the comprehensive score for each sub-index set, and choose the optimal set.

First, calculate the comprehensive score $Scor{e}_{p,n}$ of the index sets by computing the ratio of $k_n$ to $C{L}_n$.

$$Scor{e}_{p,n}=\frac{k_n}{C{L}_n}.$$

(12)

Then, select the sub-index set with the highest $Scor{e}_{p,n}$ to be the optimal index system $S_{p,Optimal}$ that takes into account completeness, independence, and simplicity given the current $p$ value, and the score is $Scor{e}_{p,Optimal}$.

Step 7: Select the global optimal index set.

Compare $Scor{e}_{p,Optimal}$ of different $p$ values. Select the optimal index system $S_{Optimal}$ through iterations.

If $Scor{e}_{p,Optimal}>Scor{e}_{p-1,Optimal}$, perform $p=p+1$ and loop through Step 3 to Step 7.

If $Scor{e}_{p,Optimal}\le Scor{e}_{p-1,Optimal}$, break the loop, where $S_{p-1,Optimal}$ is the optimal index system, which is also the $S_{Optimal}$.

3. Experiment and Results

A remote sensing satellite constellation scene with a scale of 100 satellites was selected for simulation verification. The satellite configuration adopted the walker constellation, as shown in Figure 7. The nominal constellation consists of 10 orbital planes, each of which contains 10 satellites, with an orbital height of 650 km and an orbital inclination of 98°. The observation scenes of the point target, area target, and moving target were simulated, evaluated and index screened, respectively. Note that the satellite constellations of the first two scenes were composed of single-type satellites. The satellite cluster in the moving target observation scene was composed of two types of satellites: wide-swath satellites and high-resolution satellites. Finally, the optimal evaluation index system of remote sensing satellites was obtained. All the experiments were performed on a personal computer with an Intel i7-9700 CPU.

We compared our proposed method with several quantitative and qualitative methods using the obtained index system. Qualitative methods include the Delphi [7,24] and AHP [25] methods, while quantitative methods include correlation analysis [26] and EWM [27] methods. Index systems were obtained using these different methods. By comparing their independence coefficients, the comprehensive loss rate of the contribution rate, and comprehensive scores, our proposed method was proven to be optimal.

3.1. Scene 1: Point Target Observation Task

By selecting 100 fixed targets that were randomly distributed across the globe, the point target observation scene was constructed as shown in Figure 8. The location of the targets was randomly chosen, with longitude ranging from 180° E to 180° W and latitude ranging from 50° S to 80° N. The position distribution of a group of randomly generated target points is shown in

Table A1. Target position distribution of the point target observation scene.

ID	Position	ID	Position	ID	Position	ID	Position
1	133° W, 45° S	26	51° W, 77° N	51	46° W, 45° S	76	37° W, 2° S
2	125° W, 14° N	27	163° W, 48° N	52	65° E, 19° N	77	22° W, 21° N
3	145° E, 16° S	28	106° E, 45° N	53	14° E, 39° N	78	45° W, 10° N
4	35° E, 66° N	29	32° W, 55° N	54	88° W, 27° N	79	147° E, 7° N
5	130° E, 35° S	30	146° E, 37° N	55	106° W, 44° S	80	18° E, 50° N
6	82° E, 50° N	31	59° E, 20° S	56	140° E, 52° N	81	141° E, 45° N
7	66° W, 30° N	32	121° E, 19° N	57	137° W, 35° N	82	121° W, 13° S
8	149° W, 27° S	33	11° E, 34° N	58	69° W, 54° N	83	17° E, 43° S
9	19° W, 27° N	34	166° E, 57° N	59	97° W, 64° N	84	45° E, 5° N
10	127° W, 3° N	35	158° W, 57° N	60	25° E, 69° N	85	99° E, 79° N
11	121° E, 1° N	36	24° E, 36° N	61	43° E, 67° N	86	45° E, 49° N
12	46° E, 6° S	37	80° E, 9° N	62	132° W, 19° S	87	12° W, 12° S
13	31° E, 20° S	38	17° E, 47° N	63	173° E, 49° N	88	24° E, 26° S
14	36° E, 3° S	39	124° E, 31° N	64	106° E, 17° N	89	160° W, 13° N
15	158° W, 8° S	40	69° E, 18° S	65	27° W, 69° N	90	68° W, 69° N
16	32° E, 40° S	41	50° W, 61° N	66	1° W, 58° N	91	152° W, 25° N
17	112° E, 13° N	42	180° W, 74° N	67	124° W, 40° N	92	45° E, 24° S
18	20° E, 17° S	43	168° E, 29° N	68	136° E, 9° S	93	7° E, 34° S
19	42° E, 51° N	44	170° W, 43° S	69	117° W, 11° N	94	39° W, 29° N
20	134° E, 40° N	45	69° E, 56° N	70	31° W, 14° S	95	65° W, 3° N
21	131° E, 39° S	46	124° E, 54° N	71	54° W, 6° N	96	34° W, 45° N
22	17° E, 50° S	47	134° E, 24° S	72	125° E, 34° N	97	151° E, 33° N
23	53° E, 7° S	48	170° W, 31° N	73	154° E, 42° S	98	43° W, 30° N
24	134° E, 46° N	49	90° W, 49° S	74	66° E, 22° S	99	83° W, 23° N
25	50° W, 37° S	50	11° W, 79° N	75	141° E, 21° S	100	145° W, 63° N

. The simulation started at 0:00 on 1 January 2021 and lasted for 1 day.

The system parameters and the location of the point targets were chosen at random to create 500 samples. System parameters consisted of the satellite constellation orbit configuration parameters and the satellite payload parameters. The ranges of the parameters are listed in

Table 1. The ranges of the parameters.

Type	Parameter	Symbol	Range
Orbit parameter	Orbital plane number	P	[5;10;20]
	Phase factor	F	[0, P−1]
	Orbit altitude (km)	H	[500, 1000]
	Inclination (°)	i	[80, 100]
Payload parameter	View angle (°)	$\beta$	[2, 20]

.

A batch simulation was performed on the 500 randomly generated samples. The simulation results were then used to evaluate the remote sensing satellite observation effectiveness on point targets. The input parameters and the effectiveness evaluation results of the first five samples are listed in

Table 2. The input parameters and effectiveness evaluation results of the first 5 samples.

ID	Input Parameters					Effectiveness Evaluation Results
	P	F	H (km)	i (°)	$\mathit{\beta }$	T1 (s)	T2 (s)	T3 (s)	T4 (s)	T5 (s)	T6 (s)	T10	S1 (m)	S2	S3
1	20	11	585	83	13	30206	10	5391	30198	2	5378	0.22%	7.58	1553	100.00%
2	20	4	950	88	12	12985	43	3853	12971	23	3834	0.46%	11.27	2164	100.00%
3	20	3	590	95	9	30757	206	7040	30747	200	7033	0.09%	5.26	1102	100.00%
4	10	3	915	97	10	26086	12	4707	26079	1	4694	0.27%	9.02	1742	100.00%
5	20	8	727	85	4	61497	78	4132	61495	75	4127	0.03%	2.87	664	73.00%

.

The samples were then analyzed using PCA, where the respective contribution rate of the principal components was

$$[0.603,0.188,0.092,0.068,0.047,0.002,0.001,7.189\times {10}^{-6},3.372\times {10}^{-9},1.759\times {10}^{-9}].$$

The cumulative contribution rate of the first four principal components reached 95.04%, given that the initial index number was three. Iterations started with the number of indices $p$ = 3 to construct ${\mathrm{C}}_{10}^p$ of different sets of index layer elements. For each index combination, the resulted independence coefficients $k$, the comprehensive loss rate of contribution rate $CL$, and the comprehensive score are shown in Figure 9. Noticeably, when $p$ = 4, the index set achieved the optimal value, where the comprehensive score was 8.87, $k$ = 0.2961, and $CL$ = 0.0334. In contrast, when $p$ = 3, although the independence of the optimal index set was higher, more contribution rate was sacrificed; when $p$ = 5, the contribution loss was reduced but yet its independence was lower. As a result, the optimal index set when $p$ = 4 was the global optimal solution that considered the principle of completeness, independence, and simplicity. The indices included in the optimal index set were T1, T4, S1, and S2.

Concerning the observation efficiency of point targets, we compared several different index system construction methods; the results are shown in

Table 3. Comparison results of different index sets.

Method Type	Method	Index Set	$\mathit{k}$	$\mathit{C}\mathit{L}$	$\mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
Qualitative	Delphi [7]	[T1, T2, T3, T6, S2]	0.0023	0.7615	0.0031
	Delphi [24]	[T1, T3, T6, T10, S1, S3]	0.0023	0.7332	0.0032
	AHP [25]	[T1, T6, T10, S1, S2, S3]	0.1076	0.7419	0.1450
Quantitative	Correlation analysis [26]	[T1, T2, S2, S3]	0.4203	0.7345	0.5722
	EWM [27]	[T1, T4, T10, S2]	0.0382	0.0333	1.1478
	ICCLR (Ours)	[T1, T4, S1, S2]	0.2961	0.0334	8.8653

. In terms of qualitative methods, the k values are all less than 0.11, indicating that the index systems contained redundant indices and hence their independence are lower. In addition, all CL values are higher than 0.7, indicating an insufficient level of completeness. With respect to quantitative methods, the independence of the correlation analysis screening results ($k$ = 0.4203) is better, but the completeness is inadequate ($CL$ = 0.07345). On the contrary, the screening results of the entropy weight method have better completeness ($CL$ = 0.0333) but insufficient independence ($k$ = 0.0382). By comparison, the independence of the obtained index system using the proposed method is slightly lower than that of the correlation analysis method, the completeness is basically the same as that of the entropy weight method, and the comprehensive score is higher than all other methods mentioned above. Hence, our proposed method is able to obtain an index set that has better completeness, independence, and simplicity.

Consequently, we obtained the optimal effectiveness observation index set for remote sensing satellites on point targets, which included indices of maximum revisit time, maximum observation interval, average GSD, and observation frequency.

3.2. Scene 2: Area Target Observation Task

To construct the area target observation scene, China and its surrounding areas were selected. The simulation was next carried out using the grid method, as shown in Figure 10. The detailed target position generated by the grid method is shown in

Table A2. Grid position distribution of the area target observation scene.

ID	Position	ID	Position	ID	Position	ID	Position
1	131° E, 47° N	26	119° E, 15° N	51	107° E, 35° N	76	95° E, 39° N
2	131° E, 43° N	27	115° E, 51° N	52	107° E, 31° N	77	95° E, 35° N
3	127° E, 51° N	28	115° E, 47° N	53	107° E, 27° N	78	95° E, 31° N
4	127° E, 47° N	29	115° E, 43° N	54	107° E, 23° N	79	95° E, 27° N
5	127° E, 43° N	30	115° E, 39° N	55	107° E, 19° N	80	95° E, 23° N
6	127° E, 39° N	31	115° E, 35° N	56	103° E, 51° N	81	91° E, 47° N
7	127° E, 35° N	32	115° E, 31° N	57	103° E, 47° N	82	91° E, 43° N
8	127° E, 31° N	33	115° E, 27° N	58	103° E, 43° N	83	91° E, 39° N
9	123° E, 51° N	34	115° E, 23° N	59	103° E, 39° N	84	91° E, 35° N
10	123° E, 47° N	35	115° E, 19° N	60	103° E, 35° N	85	91° E, 31° N
11	123° E, 43° N	36	115° E, 15° N	61	103° E, 31° N	86	91° E, 27° N
12	123° E, 39° N	37	111° E, 51° N	62	103° E, 27° N	87	87° E, 47° N
13	123° E, 35° N	38	111° E, 47° N	63	103° E, 23° N	88	87° E, 43° N
14	123° E, 31° N	39	111° E, 43° N	64	103° E, 19° N	89	87° E, 39° N
15	123° E, 27° N	40	111° E, 39° N	65	99° E, 51° N	90	87° E, 35° N
16	123° E, 23° N	41	111° E, 35° N	66	99° E, 47° N	91	87° E, 31° N
17	119° E, 51° N	42	111° E, 31° N	67	99° E, 43° N	92	87° E, 27° N
18	119° E, 47° N	43	111° E, 27° N	68	99° E, 39° N	93	83° E, 43° N
19	119° E, 43° N	44	111° E, 23° N	69	99° E, 35° N	94	83° E, 39° N
20	119° E, 39° N	45	111° E, 19° N	70	99° E, 31° N	95	83° E, 35° N
21	119° E, 35° N	46	111° E, 15° N	71	99° E, 27° N	96	83° E, 31° N
22	119° E, 31° N	47	107° E, 51° N	72	99° E, 23° N	97	79° E, 39° N
23	119° E, 27° N	48	107° E, 47° N	73	95° E, 51° N	98	79° E, 35° N
24	119° E, 23° N	49	107° E, 43° N	74	95° E, 47° N	99	79° E, 31° N
25	119° E, 19° N	50	107° E, 39° N	75	95° E, 43° N	100	75° E, 35° N

. The simulation started at 0:00 on 1 January 2021 and lasted for 5 days.

Similarly, 500 samples were randomly generated whose value ranges were the same as in Section 3.1; see Table 1 for details. Subsequently, batch simulation and effectiveness indices calculation were carried out.

Table 4. The input parameters and effectiveness evaluation results of the first 5 samples.

ID	Input Parameters					Effectiveness Evaluation Results
	P	F	H (km)	i (°)	$\mathit{\beta }$	T1 (s)	T2 (s)	T3 (s)	T4 (s)	T5 (s)	T6 (s)	T8 (s)	T9 (s)	T10	S1 (m)	S2	S3
1	10	4	746	84	16	15394	496	4456	15376	478	4437	15320	12589	42.64%	11.86	34	100.00%
2	5	2	872	91	13	9113	291	4790	9083	279	4773	8832	8620	35.99%	11.19	50	100.00%
3	5	1	994	86	7	43502	308	8104	43495	304	8093	41615	14830	13.66%	6.82	29	100.00%
4	10	7	919	98	4	59206	1030	14734	59198	1022	14728	39211	36346	3.43%	3.60	11	100.00%
5	20	2	711	93	7	30068	757	10573	30056	753	10566	27712	22531	6.62%	4.89	19	100.00%

provides the input parameters and effectiveness evaluation results of the first five samples.

We analyzed the samples with PCA, and the respective contribution rate of the principal components was

$$[0.576,0.186,0.099,0.066,0.044,0.019,0.009,0.001,0.001,1.937\times {10}^{-7},3.314\times {10}^{-10},1.975\times {10}^{-10}]$$

The cumulative contribution rate of the first five principal components reached 96.98%. Hence, the iteration began with = 4. As shown in Figure 11, we calculated the independence coefficient $k$, the comprehensive loss rate of the contribution rate $CL$, and the comprehensive score for each index combination. As we can see, when $p$ = 4, the index set achieved the optimal values, where the comprehensive score was 5.17, $k$ = 0.3229, and $CL$ = 0.0624. By comparison, when $p$ = 5, the contribution loss rate was almost the same as when $p$ = 4, but the independence was lower; whereas when $p$ = 6, even though the loss in the contribution rate decreased, the independence difference was obvious. As a consequence, the global optimal solution was the optimal index set when $p$ = 4. The optimal index set included four indices, which were T1, S1, S2, and S3.

Regarding the observation efficiency of area targets, we compared several different index system construction methods, and the results are shown in

Table 5. Comparison results of different index sets.

Method Type	Method	Index Set	$\mathit{k}$	$\mathit{C}\mathit{L}$	$\mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
Qualitative	Delphi [7]	[T9, T10, S2]	0.0279	0.1004	0.2783
	Delphi [24]	[T1, T3, T6, T10, S1, S3]	0.0008	0.0941	0.0086
	AHP [25]	[T1, T9, T10, S1, S2, S3]	0.0821	0.0621	1.3215
Quantitative	Correlation analysis [26]	[T1, T2, S1, S3]	0.1356	0.0626	2.1672
	EWM [27]	[T2, T5, T10, S1, S2]	0.2916	0.1016	2.8689
	ICCLR (Ours)	[T1, S1, S2, S3]	0.3229	0.0624	5.1730

. In respect of qualitative methods, all $k$ values are less than 0.09, suggesting that the index systems are of weaker independence as they contained redundant indices. Additionally, all $CL$ values are lower than 0.1, indicating a sufficient level of completeness. When comparing the screening results of quantitative methods to those of qualitative methods, the correlation analysis method and the entropy weight method not only significantly improved the independence of the screening results but also have a similar level of completeness. In contrast, the index set obtained using the proposed method in this paper is superior to the other methods mentioned above in terms of independence and completeness.

Eventually, the optimal effectiveness observation index set for remote sensing satellites on area targets was obtained, which contained maximum revisit time, average GSD, observation frequency, and coverage percentage.

3.3. Scene 3: Moving Target Observation Task

The satellite constellation employed in the moving target observation scene was composed of wide-swath satellites and high-resolution satellites. They collaborated to conduct moving target observation tasks. The main parameters of the satellite constellation are shown in

Table 6. Parameters of the satellite constellation.

Type	Parameter	Value
Orbit	Number of satellites	100
	Number of orbital planes	10
	Phase factor	1
	Semimajor axis (km)	7021
	Orbital eccentricity	0
	Orbital inclination (°)	98
Attitude	Attitude maneuver angle range (°)	[–45, 45]
	Attitude maneuver angular velocity range (°/s)	[–1, 1]
Payload	View angle of wide-swath satellites (°)	10
	View angle of high-resolution satellites (°)	2

.

As Figure 12 illustrates, we selected some ship targets in the sea area of the Asia-Pacific region to construct the moving target observation scene. Every ship target was initially located in a 220 km × 220 km area, where its position, speed, and heading were arbitrary values. The simulation started at 0:00 on a day in 2021 and lasted for 6 h.

We generated 500 samples by picking random values for the task parameters. The task parameters contained time, target, and environmental parameters that have a stronger influence on the task effectiveness [27]. The range of values for the task parameters is shown in

Table 7. The range of the sample input task parameters.

Parameter	Symbol	Range
Start time (Day)	$t$	[1, 365]
Longitude of area center (°)	$Lon$	[130, 150]
Latitude of area center (°)	$Lat$	[10, 30]
Cloud level	$N_c$	[1, 6]
Visibility level	$N_v$	[1, 6]
Number of moving targets	$N_t$	[5, 20]

.

Next, batch simulation was performed on the 500 randomly generated samples. The simulation results were utilized to complete the evaluation of the effectiveness of remote sensing satellites in observing moving targets. The input parameters and effectiveness evaluation results of the first five samples are listed in

Table 8. The input parameters and effectiveness evaluation results of the first 5 samples.

ID	Input Parameters						Effectiveness Evaluation Results
	$\mathit{t}$ (Day)	$\mathit{L}\mathit{o}\mathit{n}\mathbf{\left(}\text{°}\mathbf{\right)}$	$\mathit{L}\mathit{a}\mathit{t}\mathbf{\left(}\text{°}\mathbf{\right)}$	${\mathit{N}}_{\mathit{c}}$	${\mathit{N}}_{\mathit{v}}$	${\mathit{N}}_{\mathit{t}}$	T4 (s)	T5 (s)	T6 (s)	T12 (s)	T13 (s)	T14	T15 (s)	S2	S4	S5
1	293	139.31	17.14	2	6	10	13,873	311	4083	4785	4375	0.84%	3864	38	86.50%	79.23%
2	296	141.87	24.55	1	3	11	7768	105	2585	5953	2534	2.36%	2498	62	90.46%	77.67%
3	355	134.53	18.91	3	2	8	7845	253	2798	5732	2009	1.69%	3110	40	99.38%	90.54%
4	284	132.30	21.73	5	5	17	11445	297	3854	6575	3701	0.46%	3995	46	87.65%	58.41%
5	319	148.87	14.88	5	4	20	13316	163	4264	3697	3284	0.78%	4757	68	90.50%	65.43%

.

PCA was performed on the samples and the contribution rate of the principal components was

$$[0.413,0.255,0.138,0.072,0.044,0.030,0.020,0.014,0.006,0.006].$$

The cumulative contribution rate of the first six principal components was 95.17%. Hence, the number of indices iterated from $p$ = 5, and ${\mathrm{C}}_{10}^p$ different sets of elements for each index layer were generated. As can be seen from Figure 13, when $p$ = 5, the index set had the optimal values, in which the comprehensive score was 3.60, the independence coefficient was 0.2702, and the comprehensive loss rate of the contribution rate was 0.0751. Comparing the results of $p$ = 5 to $p$ = 6, the contribution loss rate was basically the same while the loss in independence was higher. By comparing the results of $p$ = 5 to $p$ = 7, albeit the contribution loss rate had decreased, the difference in independence was significant. From this experiment, the optimal index set was obtained when $p$ = 5, which was a globally optimal solution that considered the principle of completeness, independence, and simplicity. The optimal index set included indices T5, T12, T15, S4, and S5.

As for the observation efficiency of moving targets, we compared different index system construction methods; the comparison results are shown in

Table 9. Comparison results of different index sets.

Method Type	Method	Index Set	$\mathit{k}$	$\mathit{C}\mathit{L}$	$\mathit{S}\mathit{c}\mathit{o}\mathit{r}\mathit{e}$
Qualitative	Delphi [7]	[T4, T5, T6, T14, S2]	0.0683	0.1375	0.4969
	AHP [7]	[T12, T13, T14, T15, S4, S5]	0.0747	0.0899	0.8306
Quantitative	Correlation analysis [7]	[T14, T15, S4, S5]	0.2613	0.1029	2.5406
	EWM [27]	[T4, T6, T12, T13, T14, S2, S5]	0.0012	0.0612	0.0189
	ICCLR (Ours)	[T5, T12, T15, S4, S5]	0.2702	0.0751	3.5972

. In terms of qualitative methods, all $k$ values are lower than 0.08, meaning that the index systems are of lower independence. Meanwhile, the $CL$ values are around 0.1, where their completeness is acceptable. With regard to quantitative methods, the screening results of correlation analysis are of a higher level of independence and completeness. On the other hand, even though the entropy weight method produced an index system with the highest completeness, it sacrificed independence, resulting in the lowest comprehensive score. In contrast, although the index set obtained by the proposed method in this paper has slightly lower completeness than that of the entropy weight method, not only is its independence better than the other methods, but it also has the highest comprehensive score.

Lastly, the optimal effectiveness evaluation index set of remote sensing satellites in observing the moving targets was obtained. The optimal index set included indices of minimum observation interval, average discovery response time, average tracking interval, discovery probability, and identification probability.

4. Discussion

Aiming to overcome the problem that the evaluation index system cannot take into consideration all the construction principles, this paper proposed an index system construction method based on quantitative analysis. After screening, an optimal index system that satisfies the principle of completeness, hierarchy, measurability, independence, and simplicity was established, as shown in Figure 14.

The index system consisted of three layers, which were the system layer, task layer, and index layer, respectively. Without considering the relatively independent indices of spectral resolution and radiometric resolution, the number of indices in the index system decreased from 32 to 13. As compared with the mainstream index system construction and screening methods, it can be seen that the effect of quantitative methods is generally better than qualitative methods. Among them, the independence of the screening results of correlation analysis is better, and the completeness of the screening results of the entropy weight method is better. By comparison, our proposed method can effectively select the least number of indices that conform to the five principles from numerous redundant indices. As compared with conventional research, the index system given in this paper not only improves the completeness but also fulfills the requirements of independence.

Apart from remote sensing satellite effectiveness evaluation, our proposed method is also applicable to other types of satellite as well as other evaluation problems. Unlike the traditional methods, the method proposed can evaluate the index system and hence provide a basis for the screening of an optimal index system. Nevertheless, this method relies on the calculation results of a large number of samples, which in turn increases the requirements for the simulation model and computational resources. In addition, the index screening results are affected by the sample space. The dimension of the sample space and the value range of each dimension will influence the final results.

Future research will be undertaken to include the following areas. Firstly, the system and task parameters that will affect the effectiveness indices can be obtained through data mining. This can improve the quality of the index system and reduce the consumption of computing resources. By using data mining to find out the main influencing factors, the granularity and the sample space can be refined. Meanwhile, the resource consumption of the nonessential elements can be reduced, which hence provides a better index system. Secondly, the collaborative optimization of the index system construction method and the effectiveness evaluation method—the index system must be combined with the evaluation method to give the system’s comprehensive score. Thus, through collaborative optimization, the design process and application process of the index system can be merged to produce a better optimal evaluation criterion. Finally, other effectiveness aspects of the satellite system also deserve further attention, such as the availability and capability of communication links, data processing capability, security, and reliability.

5. Conclusions

The effectiveness evaluation of remote sensing satellites has attracted considerable interest due to their advantages in global coverage, high temporal resolution, and spatial resolution. The construction process of the evaluation index system is a crucial part of effectiveness evaluation research. However, the process not only contains subjective qualitative factors but also has difficulties in considering all principles. To solve these problems, we have proposed an index system construction method that takes all principles into account. The advantages of the proposed method are summarized as follows.

• The construction method we proposed combines qualitative and quantitative methods. It not only complies with mainstream research, but also considers all principles, including completeness, hierarchy, measurability, independence, and simplicity.
• Aside from building an index system from scratch, the proposed method can also supplement the index system based on some artificially specified indices. At the same time, it can also act as an evaluation mechanism of the index system, mainly used to find out the subjective problems and cognitive defects of the index system that are qualitatively formulated by experts. Suggestions can then be given for further improvement.
• According to the specific research scope of the system and task environment included in the evaluation object, the proposed method has been able to provide the most suitable index system for the problem depending on the quantitative calculation results. When compared to the current inflexible index system, the flexibility of the obtained index system has also improved.

To enhance the construction process of the index system and its applicability, future research can be carried out on the following three aspects:

• The perfection of simulation models such as the ground data processing model, orbital model, cloud model, and visibility model.
• The collaborative optimization of index system construction and evaluation methods.
• The supplement of other aspects of satellite effectiveness.