A Remote Sensing Image Quality Interpretation Scale Characterization Method Based on the TTP Criterion

Abstract

Accurate grading of remote sensing image interpretation is crucial for improving image classification and screening efficiency. Through extensive research, the General Image Quality Equation (GIQE) based on the National Imagery Interpretability Rating Scale (NIIRS) has been developed. However, poor reliability and low accuracy issues remain due to the failure to consider human visual characteristics. This paper introduces the Target Task Performance (TTP) criterion as a key parameter to reflect the cascading degradation factors of human visual perception characteristics and system imaging links, which improves the reliability of the model. A New Optimized Remote Sensing Image Quality Equation (NORSIQE), which effectively predicted the interpretability of image information, is constructed. Using 200 sets of test data, the quantitative relationship between key parameters (GSD, TTP, and SNR) of the NORSIQE and the subjective NIIRS level is obtained by a least squares regression fit, and the determination coefficient of the model is as high as 0.916. The model is evaluated for accuracy using 120 sets of validation data, showing an 87% improvement compared to the GIQE4. This method provides theoretical support for the development of new methods for remote sensing image quality evaluation and the design of payloads for remote sensing imaging systems.

1. Introduction

With the continuous updating of remote sensing technology, the number of satellites has increased and the ability of data acquisition has been greatly improved. However, in the face of massive image data, it is necessary to evaluate image quality and interpretability in order to effectively improve the screening efficiency. Therefore, it is of great practical significance to study the evaluation method of the remote sensing image quality interpretation scale [1,2,3,4,5,6,7,8,9,10,11,12,13,14].

The National Imagery Interpretability Rating Scale (NIIRS) [15,16,17,18,19] is the most commonly used international subjective evaluation guideline for image quality. It is a task-based scale that classifies image quality into 10 levels from 0 to 9, with the degree of interpretation increasing step by step. On the other hand, the General Image Quality Equation (GIQE) [19,20,21,22,23,24,25,26,27,28] is a quality assessment model developed for the NIIRS. It uses the imaging parameters of the image itself to assess the NIIRS of the image.

There are three known versions of the GIQE: GIQE3, GIQE4 and GIQE5. The latest version, the GIQE5, was released in 2015 by the National Geospatial-Intelligence Agency (NGA) through a paper on its website [25]. However, because it only calculates a single NIIRS level for “well-enhanced” images as a function of their unenhanced parameters without considering processing details and cannot be used to predict NIIRS, the GIQE4 is still widely used [28], and the model in this paper is optimized based on the GIQE4.

The GIQE4 quantifies the degree of degradation of the image in terms of spatial resolution and sharpness, forming a mapping relationship between image imaging parameters and NIIRS grade. However, its failure to characterize the human eye’s ability to perceive image information in prediction calculations makes the model’s prediction results largely unreliable and unconvincing, as well as having the problem of low accuracy.

Researchers from different countries have conducted studies on image quality interpretation from various perspectives. For instance, Valenzuela et al. [28] conduct a comparative analysis of different versions of the GIQE, while Jingbo Bai et al. [29] study image NIIRS and GIQE for Unmanned Aerial Vehicle (UAV) intelligence and reconnaissance actions. Lin L et al. [30] establish a regression model of manual interpretation results and model prediction results, considering that the ground spatial resolution of the ZY-3 satellite is not within the applicable range of parameters of the GIQE model. Pengfei Zhao [31] improves the GIQE model by using three different Modulation Transfer Function (MTF) performance parameters.

The above-mentioned studies conducted different analytical investigations into the GIQE and optimized image quality evaluation methods from certain perspectives, but none of them combined the human eye’s ability to perceive image information with the prediction calculation of the subjective NIIRS level of experts’ judgments (abbreviated in later papers as subjective NIIRS level), resulting in prediction models that still suffer from poor reliability and low accuracy.

In response to this issue, this paper analyzes the correlation between the target task performance (TTP) criterion [32,33,34,35,36,37] and remote sensing image quality interpretation, takes the TTP criterion as a key parameter in the prediction model of the remote sensing image quality interpretation scale, and constructs a New Remote Sensing Image Quality Equation (NRSIQE). This equation can reflect the characteristics of the human visual system’s perception of image information in the prediction. On completion of regression fits and precision analysis, a New Optimized Remote Sensing Image Quality Equation (NORSIQE) is constructed by choosing a logarithmic function to represent the signal-to-noise ratio term. Compared with the GIQE4, the prediction accuracy is improved by 87%. The verification results show that this method can effectively improve the prediction reliability and accuracy of the remote sensing image quality interpretation model.

2. Deficiencies of the GIQE

The GIQE is an image quality assessment model that was developed for the NIIRS. It utilizes the system imaging parameters of the image, which are objective evaluation metrics, to evaluate the NIIRS, which is a subjective evaluation metric. The principle of the GIQE is illustrated in Figure 1. And the specific calculation formula of the GIQE4, which is widely used, is as follows:

$$\mathrm{NIIRS}=10.251-\mathrm{a}\lg (\mathrm{GSD})+\mathrm{b}\lg (\mathrm{RER})-0.656\mathrm{H}-0.334(\frac{\mathrm{G}}{\mathrm{SNR}})$$

(1)

where lg(x) = base 10 logarithmic function of x; GSD = Ground Sampling Distance (in inches); RER = Relative Edge Response; H = Edge response overshoot caused by Modulation Transfer Function Compensation (MTFC); G = sharpening filter gain; SNR = signal-to-noise ratio; a and b are constants, a = 3.32 and b = 1.559 when RER < 0.9, a = 3.16, and b = 2.817 when RER < 0.9.

As in Equation (1), the GIQE4 predicts the image quality interpretation scale from the spatial resolution, sharpness and noise of the image, which only takes into account the objective imaging quality parameters of the image and focuses on the influence of spatial resolution degradation on the image quality and interpretability. However, as a specific behavior driven by human subjective consciousness, the results of image interpretability [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19] evaluation will be affected by the perceptual characteristics of human eyes on the information contained in the image. The GIQE4 does not takes into account the ability of the human visual system to perceive image information, so it cannot accurately predict the human understanding of the information contained in the image and the judgment of the image interpretation task, which leads to the lack of reliability and accuracy of the prediction model.

To overcome the problem that the subjective visual perception factors of the observer are not reasonably considered in the GIQE4 evaluation, and then to achieve the purpose of improving the accuracy and completeness of image quality interpretation scale prediction, in this paper, we propose to introduce the TTP criterion into the remote sensing image quality equation, to reflect the effect of the perceptual characteristics of human visual image information on the human judgment of image interpretation tasks in the prediction model.

3. Theoretical Basis for the Introduction of the TTP Criterion

The quality and interpretability scale of remote sensing images are closely related to the objective image quality and human perception. In order to effectively improve the prediction accuracy of the model, it is necessary to evaluate the image quality and combine the understanding and perception characteristics of the human visual system with the information contained in the image.

The TTP criterion is a comprehensive performance evaluation index defined by the Night Vision and Electronic Sensors Director (NVESD) to describe the effectiveness of human eye-space integration in the effective spectrum [32,33]. Figure 2 illustrates the schematic diagram of the TTP criterion, where the intersection point of the imaging contrast function ${\mathrm{C}}_{tgt}{\mathrm{MTF}}_{sys}(\xi )$ (where ${\mathrm{C}}_{tgt}$ represents the target modulation contrast, ${\mathrm{MTF}}_{sys}(\xi )$ represents the Modulation Transfer Function of the system and $\xi$ represents the spatial frequency) and the Contrast Threshold Function of the human eye (${\mathrm{CTF}}_{eye}({\xi }_{eye})$) is ${\xi }_h$. This represents the limit resolution of the image interpretation task, where only the part where ${\mathrm{C}}_{tgt}{\mathrm{MTF}}_{sys}(\xi )$ is greater than ${\mathrm{CTF}}_{eye}({\xi }_{eye})$ can be perceived by human vision. All the information in the effective spectrum range can be integrated to obtain all the image information perceived by the human eye, which characterizes the spatial integration effect of the human eye. The calculation expression is:

$$\mathrm{TTP}={\displaystyle \underset{{\xi }_l}{\overset{{\xi }_h}{\int }}\sqrt{\frac{{\mathrm{C}}_{tgt}{\mathrm{MTF}}_{sys}(\xi )}{{\mathrm{CTF}}_{eye}({\xi }_{eye})}}}d\xi$$

(2)

$${\mathrm{MTF}}_{sys}(\xi )={\mathrm{MTF}}_{opt}(\xi )·{\mathrm{MTF}}_{\det }(\xi )·{\mathrm{MTF}}_e(\xi )·{\mathrm{MTF}}_{dis}(\xi )$$

(3)

$${\mathrm{CTF}}_{eye}({\xi }_{eye})=\frac{a}{b\xi e^{-c{\xi }_{eye}}\sqrt{1+0.06e^{-c{\xi }_{eye}}}}$$

(4)

$$a=1+\frac{12}{\omega (1+\frac{{\xi }_{eye}}{3})}$$

(5)

$$b=\frac{540}{{(1+\frac{0.7}{L})}^{0.2}}$$

(6)

$$c=0.3{(1+\frac{100}{L})}^{0.15}$$

(7)

where ${\xi }_l$ = integration start frequency; ${\xi }_h$ = integration cutoff frequency; ${\mathrm{MTF}}_{opt}(\xi )$ = the MTF of the optical system; ${\mathrm{MTF}}_{\det }(\xi )$ = the MTF of the detector; ${\mathrm{MTF}}_e(\xi )$ = the MTF of the electronic circuit; ${\mathrm{MTF}}_{dis}(\xi )$ = the MTF of the display; Ref. [37] ${\xi }_{eye}=\xi /\mathrm{SMAG}$ represents the spatial frequency of the human eye; SMAG = the system angular magnification; $\omega$ = the square root of the graphic area expressed in degrees; L = the brightness of the display in cd/m² [38].

Both are subject to the condition that ${\mathrm{C}}_{tgt}{\mathrm{MTF}}_{sys}(\xi )$ is greater than ${\mathrm{CTF}}_{eye}({\xi }_{eye})$. It is the contrast threshold of image information that the human eye can perceive, which is affected by spatial resolution, image area, display average brightness and system magnification. And it reflects the ability of human vision to perceive image information in different display states.

It can be seen that the TTP criterion is obtained by integrating ${\mathrm{C}}_{tgt}{\mathrm{MTF}}_{sys}(\xi )$ and ${\mathrm{CTF}}_{eye}({\xi }_{eye})$ in effective spatial frequency after comparing mappings. It can not only reflect the impact of imaging system performance, the cascade diffusion phenomenon of the remote sensing optical imaging system, the target background contrast, the overall image quality characteristics, the display brightness adjustment and other factors, but also characterize how human eyes perceive the effective information of remote sensing images. It is a comprehensive representation index of objective image quality and subjective human perception characteristics.

In view of this, the TTP criterion is introduced into the remote sensing image quality interpretation model, which can provide theoretical support to improve the reliability and accuracy of the remote sensing image interpretability prediction model.

4. New Quality Equation for Remote Sensing Images

The image quality equation is a bridge between the subjective and objective evaluation of images [12,13,14]. In this paper, the TTP criterion, as a comprehensive evaluation index, is proposed as a parameter in the new equation for remote sensing image quality. Thus, the objective and subjective evaluation factors can be effectively combined to improve the reliability and accuracy of predictions [32,33,34,35,36].

4.1. Construction of the New Model

In the GIQE4, RER represents the diffusion degree of image Edge Spread Function (ESF) and the grayscale diffusion range of the image reflects the degradation of image edge sharpness, but it is not enough to represent the overall image quality. In contrast, ${\mathrm{MTF}}_{sys}(\xi )$ covered by the TTP criterion can measure the diffusion phenomenon and image clarity in remote sensing optical imaging systems and can reflect the on-orbit imaging performance of sensors and the overall image quality more intuitively than RER [11]. And ${\mathrm{CTF}}_{eye}({\xi }_{eye})$ is the contrast threshold of image information that the human eye can perceive, which is affected by spatial resolution, image area, display average brightness and system magnification. And it reflects the ability of human vision to perceive image information in different display states [37]. In addition, the TTP criterion [32,33,34,35,36] is directly used as the independent variable of the remote sensing image quality equation, which has characterized the role of MTFC to a certain extent. Meanwhile, due to the close correlation between edge overshoot H, noise gain G and modulation transfer function compensation (MTFC), this paper directly extracts the MTF of the image, which has to some extent taken into account the influence of H and G. Therefore, H and G caused by MTFC are no longer used as independent variables in image quality equations, which can reduce the complexity of the model.

Meanwhile, the TTP criterion considers all spatial frequency information that can be perceived by human eyes on the display, reflecting the spatial integral effect of the human vision system. Therefore, the TTP criterion is introduced into the image quality interpretation scale prediction model, which can realize the precision interpretation of conventional aperiodic remote sensing image quality.

Furthermore, the quality and interpretability scale of remote sensing images are closely related to the detail definition of the images. While spatial resolution reflects the detailed information contained in an image, noise has a significant impact on image sharpness and contrast. Therefore, GSD and SNR continue to be regarded as the key parameters in the NRSIQE in the scale estimation of remote sensing image quality.

In view of the above analysis, the specific expression of the NRSIQE is shown in Equation (8).

$$\mathrm{NIIRS}=\mathrm{c}1+\mathrm{c}2 \lg \left(\mathrm{GSD}\right)+\mathrm{c}3 \lg \left(\mathrm{TTP}\right)+\frac{\mathrm{c}4}{\mathrm{SNR}}$$

(8)

where lg(x) = base 10 logarithmic function of x; GSD = Ground Sampling Distance (in inches); TTP = information acquisition performance based on the integration of image contrast and human eye contrast threshold transfer function in the frequency domain; SNR = signal-to-noise ratio; c1, c2, c3 and c4 = undetermined coefficients.

In order to determine the quantitative relationship between the key parameters (GSD, TTP and SNR) in the NRSIQE and the subjective NIIRS level, the imaging parameters of 200 groups of remote sensing image slices from three different sensors and different times and places, and the corresponding subjective NIIRS level (which is evaluated by more than five experts before the median level is determined) are used as the test data set.

Table 1. Image raw data information.

Type	Date	Time	Location	Spatial Resolution	Subjective NIIRS Level
XX_1	2019.07.19	09:24:06	E_118°07′14″, N_32°27′21″	6.80 inches	4
XX_1	2019.06.24	16:18:30	E_119°46′41″, N_31°22′34″	25.78 inches	5
XX_2	2013.09.15	09:00:00	E_117°12′17″, N_23°33′42″	6.99 inches	7
XX_2	2018.07.05	08:22:09	E_117°39′45″, N_24°33′42″	7.22 inches	7
XX_2	2021.03.23	15:22:06	E_120°06′07″, N_30°57′32″	2.72 inches	7
XX_3	2018.02.08	16:36:26	E_120°06′07″, N_30°57′32″	3.27 inches	5
XX_3	2019.06.14	13:50:50	E_120°06′04″, N_30°06′04″	1.46 inches	6
XX_3	2020.02.11	13:35:23	E_123°03′01″, N_41°52′10″	4.20 inches	6

shows part of the image information in the test set, and the sample of remote sensing image slices is shown in Figure 3.

Based on the sample data corresponding to the test data set, the regression model corresponding to the NRSIQE is established by using the least squares method, and the fitted results obtained are shown in Figure 4.

In the above figure, the blue solid line represents the prediction result of the regression-fitted model corresponding to the NRSIQE, and the orange dotted line represents the subjective NIIRS level. Evidently, the predicted value of the NRSIQE obtained by fitting is in good agreement with the subjective NIIRS level, and the corresponding model expression is:

$$\mathrm{NIIRS}=3.3341-2.729 \lg \left(\mathrm{GSD}\right)+1.3091 \lg \left(\mathrm{TTP}\right) -\frac{0.2741}{\mathrm{SNR}}$$

(9)

The determination coefficient of the model is ${\mathrm{R}}^2=0.421$, and the value p = 9.67 × 10⁻⁷³ is rejected, which proves that the fit of the equation is good and the regression coefficient is significant.

In order to further improve the reliability of the regression model, consider improving the stability of the data without changing the nature and relationship of the data. On the basis of Equation (8), the logarithmic function is chosen to express the SNR term, which not only improves the stability of the data, but also reduces the redundancy of the model. A NORSIQE with a unified form of operation is built. The specific expression is:

$$\mathrm{NIIRS}=\mathrm{c}1+\mathrm{c}2 \lg \left(\mathrm{GSD}\right)+\mathrm{c}3 \lg \left(\mathrm{TTP}\right)+\mathrm{c}4 \lg \left(\mathrm{SNR}\right)$$

(10)

The meaning of each parameter in the above formula is the same as that in Equation (8).

Similarly, in order to clarify the values of c1, c2, c3 and c4, the regression model corresponding to the NORSIQE was fitted with the same method, and the expression is as follows:

$$\mathrm{NIIRS}=4.6366-3.2058 \lg \left(\mathrm{GSD}\right)+0.5996 \lg \left(\mathrm{TTP}\right)+3.5611 \lg \left(\mathrm{SNR}\right)$$

(11)

The determination coefficient of the model is ${\mathrm{R}}^2=0.916$, and the value p = 6.54 × 10⁻¹⁰⁶ is rejected.

The coefficients of the NRSIQE and NORSIQE and the corresponding regression analysis indicators were compared and analyzed, and the results are shown in

Table 2. NRSIQE and NORSIQE coefficients for GSD in inches.

Equation	c1	c2	c3	c4 f(x)	R2	p	RMSE
NRSIQE	3.3341	−2.7291	1.3091	−0.2741/SNR	0.793	9.67 × 10−73	0.421
NORSIQE	4.6366	−3.2058	0.5996	3.5611 lg(SNR)	0.916	6.54 × 10−106	0.204

.

By comparing the determination coefficient (${\mathrm{R}}^2$) and Root Mean Square Error (RMSE) of the two regression models, it can be seen that the NORSIQE has higher regression significance and reliability, and its prediction results can form a good consistent correlation with the subjective NIIRS level corresponding to the image samples in the remote sensing image test data set.

4.2. Verification of the New Model

In order to verify the prediction accuracy and validity of the NORSIQE, the imaging parameters of another 120 sets of remote sensing image slices from four different sensors and different times and places, and their corresponding subjective NIIRS level were used as validation data set. The absolute value of the difference between the predicted results of the GIQE4, NRSIQE, and NORSIQE and the subjective NIIRS level, is shown in Equation (12), expressed as ∆NIIRS.

$$\Delta \mathrm{NIIRS}=\left|{\mathrm{NIIRS}}_{prediction}-{\mathrm{NIIRS}}_{subjective}\right|$$

(12)

where ${\mathrm{NIIRS}}_{prediction}$ = the NIIRS level predicted by each model (the GIQE4, NRSIQE, and NORSIQE), ${\mathrm{NIIRS}}_{subjective}$ = the subjective NIIRS level.

The comparative analysis results of prediction accuracy obtained are shown in Figure 5.

According to ∆NIIRS corresponding to the three models in Figure 5a, it can be seen that the prediction result of the GIQE4 has the highest error value with respect to the subjective NIIRS level, and has no stability. The NRSIQE prediction results have the most stable error value with respect to the subjective NIIRS level. The NORSIQE’s prediction results have the least deviation from the subjective NIIRS level. According to Figure 5b, it is more intuitively reflected that the NORSIQE has a higher prediction accuracy compared with the NRSIQE.

On the basis of the corresponding interpretation task of this study, the data samples with the subjective NIIRS level of 4, 5, 6, 7 and 8 are screened, and 20 groups of samples are used as validation data respectively. Three models (GIQE4, NRSIQE and NORSIQE) are used to interpret and predict the remote sensing image quality of different subjective NIIRS, and the calculation deviation ∆NIIRS of each model is calculated. The prediction accuracy of the three models for different interpretation tasks was compared, and the results are shown in Figure 6.

As can be seen from Figure 6, for the quality prediction of remote sensing images with different interpretation tasks, the NORSIQE has fewer prediction errors and shows better stability than the GIQE4 and NRSIQE.

In order to quantitatively compare the prediction accuracy of the above three models, the RMSE corresponding to the three models was calculated based on the subjective NIIRS level, as shown in

Table 3. RMSE of GIQE4, NRSIQE and NORSIQE.

Error Indicator	Subjective NIIRS Level	GIQE4	NRSIQE	NORSIQE
RMSE	4	2.919	0.302	0.164
	5	2.205	0.340	0.234
	6	2.346	0.343	0.259
	7	1.315	0.379	0.271
	8	1.339	1.039	0.324
$\overline{\mathrm{RMSE}}$	----	2.025	0.481	0.250

.

According to the data in the table, for the remote sensing image sample data of different interpretation tasks in the validation data set, the average prediction deviation of the NORSIQE is 0.231 smaller than that of the NRSIQE, and 1.775 smaller than that of the GIQE4.

For the prediction results of the three models, the statistics ∆NIIRS ≤ 0.1, 0.1 < ∆NIIRS ≤ 0.2, 0.2 < ∆NIIRS ≤ 0.3, 0.3 < ∆NIIRS ≤ 0.5 and ∆NIIRS > 0.5 corresponding to the number of image slices in the validation set, and the obtained percentage results are shown in Figure 7.

The above figure shows that 64% of the prediction deviations of the NORSIQE constructed in this paper are within 0.1 in the validation data set, and more than 95% of ∆NIIRS ≤ 0.5, enhanced by 44% and 87% compared to the prediction accuracy of NRSIQE and GIQE4, respectively. It can be seen that the prediction results of NORSIQE can show some validity for different tasks of interpreting remote sensing image information.

Meanwhile, the accuracy ratio of the model proposed in this paper, along with the MTF-Nyquist model, MTF-50 model, and MTF-Area model proposed in reference [31], was statistically analyzed according to the ∆ NIIRS interval mentioned above, and the results are shown in Figure 8.

From the figure, it can be seen that the MTF-50 model proposed in reference [31] has the best prediction accuracy of 50% for situations with ∆ NIIRS ≤ 0.1. However, in comparison, the prediction accuracy of the NORSIQE model proposed in this paper exceeds this by 14%, which has certain advantages.

The above comprehensive validation results show that introducing the TTP criterion to construct the remote sensing image quality equation and the logarithmic function which has been used to process the SNR parameters can effectively improve the regression significance and prediction accuracy of the remote sensing image quality equation. Therefore, this paper uses the NORSIQE as a model expression to quantitatively describe the interpretability of remote sensing image information, as it characterizes the perceptual properties of the human visual system on the information contained in remote sensing images while comprehensively describing the image diffusion and overall image quality degradation. It thus constitutes a new scale characterization method of remote sensing image quality interpretation based on the TTP criterion, which improves the accuracy by 87% compared with the GIQE4, and it also effectively improves the reliability of the prediction model.

5. Conclusions

In view of the unreliability and inaccuracy of the NIIRS predicted by the existing GIQE4 model due to its inability to represent human visual perception features, the TTP criterion is introduced into the image quality equation as a key parameter in this paper, which not only represents the objective image quality of remote sensing images, but also reflects the spatial integration effect of the human visual system. In this paper, a regression fit and accuracy assessment of the model are performed using a test set and validation set consisting of 320 data. Through comparative analysis, a NORSIQE model with a decision coefficient of up to 0.916 is established. Compared to the GIQE4, its prediction accuracy improves by 87% (when ∆NIIRS ≤ 0.5). This method provides a new way to evaluate the quality of remote sensing images.

In the future, we will start from the full application of remote sensing photoelectric imaging, considering the physical effects of imaging, combined with innovative technologies, to further improve the intelligence and automation of remote sensing image interpretation task prediction.