Feature Evaluation and Comparison in Radar Emitter Recognition Based on SAHP

: In the ﬁeld of radar emitter recognition, with the wide application of modern radar, the traditional recognition method based on typical ﬁve feature parameters cannot achieve satisfactory recognition results in a complex electromagnetic environment. Currently, many new feature extraction methods are presented, but few approaches have been applied for feature evaluation or performance comparison. To deal with this problem, a feature evaluation and selection method was proposed based on set pair analysis (SPA) theory and analytic hierarchy process (AHP). The main idea of this method is to use SPA theory to solve problems regarding the construction of the decision matrix based on AHP, as it relies heavily on expert’s subjective experience. The aim was to improve the objectivity of the evaluation. To check the effectiveness of the proposed method, six feature parameters were selected for a comprehensive performance evaluation. Then, the convolutional neural network (CNN) was introduced to validate the recognition capability based on the evaluation results. Simulation results demonstrated that the proposed method could achieve the feature analysis and evaluation more reasonably and objectively.


Introduction
In modern warfare, radar emitter recognition is an important part of an electronic support system (ESM). The recognition level has become an important symbol by which to measure the technology level of the reconnaissance equipment [1,2]. With the rapid development of new radar technology and the wide use of modern radar, the electromagnetic environment is becoming more and more complex. The traditional recognition methods are based on the typical five parameters, which include radio frequency (RF), pulse repetition interval (PRI), pulse width (PW), angle of arrival (AOA), and pulse amplitude (PA). However, the traditional recognition methods cannot meet the needs of emitter recognition in current radar signal environment with the attributes of high density, high complex waveforms, and broadband spectrum agility [3][4][5]. Therefore, many experts and scholars have made great efforts to find new features to compensate for the shortcomings of the typical five parameters. In recent studies, many new features have been successfully applied in emitter recognition, such as entropy [6], resemblance coefficient [7], dual spectrum features [8,9], fingerprint characteristics [10], and so on. However, in face of multiple features, how to evaluate and choose these features are the urgent problems to be solved in practical emitter recognition.
Currently, many performance evaluation methods on emitter signals recognition have been successfully applied, such as the technique for order preference by similarity to ideal solution (TOPSIS) [11], intuitionistic fuzzy network [12], the relative entropy, and elimination choice translation methods [13]. These methods can obtain a reasonable evaluation order of the effectiveness evaluation on emitter signals recognition. However, there is little research on the evaluation of emitter signal features. Therefore, it becomes important to seek the method for emitter signal feature evaluation. The analytic hierarchy process (AHP) [14][15][16] is a systematic analysis method applied on feature evaluation extensively. It can transform the unstructured description of a complex problem to a structured description effectively. Meanwhile, it is a powerful tool to study the decisionmaking problems of complex systems with multiple objectives, criteria, and attributes. Zhu et al. [17] proposed a method of radar emitter feature evaluation based on the AHP theory. In [17], the decision matrix is constructed by the expert scoring to obtain the evaluation order of the emitter signal features. Xu et al. [18] proposed a radar emitter signal feature evaluation method based on AHP-interval-TOPSIS. The purpose of this method is to minimize the subjectivity of the decision matrix construction. These evaluation methods based on AHP provide a new perspective for feature evaluation. However, both of them have two common problems. One is that the method relies on the expert's subjective experience excessively in the processing of the decision matrix construction. The second is the feature parameters uncertainty of the measured radar emitter signals. Due to the complex electromagnetic environment and types of jammings, the measured feature parameters always have distortions compared to their true values. These problems bring about many difficulties to feature evaluation in emitter recognition.
Aiming at the problems of feature evaluation methods based on the AHP mentioned above, a new feature evaluation method is proposed combining set pair analysis (SPA) [19,20] and AHP in this paper. This approach adopts SPA to construct the decision matrix, which makes full use of the advantages of SPA theory in describing the certainty and uncertainty of the set. It can improve the objectivity of the decision matrix and scientific index weight setting. In order to validate the effectiveness of the proposed method, six features were selected for comprehensive evaluation. Then, the CNN classifier was introduced to check the recognition capability based on the evaluation results. Simulation results show that the proposed method is effective and feasible. It can achieve the feature evaluation more objectively.
The rest of this paper is organized as follows. In Section 2, the algorithm of the feature evaluation based on AHP is given, and the feature evaluation model is constructed in terms of the practical application of emitter recognition. In Section 3, the principle of SPA theory is introduced to construct the comprehensive evaluation of the decision matrix. Section 4 features comprehensive evaluations that are carried out based on the set pair analytic hierarchy process (SAHP). Then, the CNN classifier is utilized to validate the recognition performance based on the evaluated features. Finally, conclusions are drawn in Section 5.

Algorithm of Feature Evaluation by AHP
The main idea of the feature evaluation algorithm based on AHP is using the AHP to establish the feature evaluation model, then constructing the judgment matrix to determine the weight of each of the features to be evaluated. The specific implementation steps of the feature evaluation method by AHP are given as below.
Step 1: Build a feature evaluation model. In this paper, the feature evaluation model includes three layers: the target layer, criteria layer, and index layer. The target layer is the feature to be evaluated. The criteria layer is the criterion for evaluating this feature. According to the practical application of emitter recognition, the evaluated feature needs to have the advantages of simple operation, high accuracy, and strong noise immunity. Thus, complexity, accuracy, and robustness are taken as the evaluation criterions in the criteria layer. In the index layer, seven indexes are selected to describe the feature-specific performance. These indexes are time complexity, spatial complexity, recognition accuracy, separation between clusters, compactness within cluster, noise immunity, and stability.
A model of the feature evaluation system can be show in Figure 1.
Electronics 2021, 10, x FOR PEER REVIEW 3 of 12 time complexity, spatial complexity, recognition accuracy, separation between clusters, compactness within cluster, noise immunity, and stability. A model of the feature evaluation system can be show in Figure 1. Step 2: Construct the judgment matrix in each layer. The judgment matrix is constructed according to the Saaty-relative importance hierarchy table [17,21]. The Saaty-relative importance hierarchy table is shown in Table 1. Indictator i and indicator j are equally important 3 Indictator i is slightly more important than indicator j 5 Indictator i is more important than indicator j 7 Indictator i is obviously more important than indicator j 9 Indictator i is absolutely more important than indicator j 2, 4, 6, 8 Between the hierarchy values mentioned above Step 3: Calculate the weight of the judgment matrix, and then check the consistency of the judgment matrix.
For checking the consistency, the compatibility index (CI), random consistency index (RI), and compatibility ratio (CR) proposed in [17] have been introduced.
where max  denotes the maximum eigenvalue of the judgment matrix, n denotes the order of the judgment matrix, CI is the compatibility index, and RI is the random consistency index. According to [21], the relationship between n and RI is given as below: when n = 2, RI = 0; when n = 3, RI = 0.58; when n = 4, RI = 0.96.
The rules of consistency testing are as follows. If CR < 0.1, the judgment matrix can be accepted. If CR ≥ 0.1, the judgment matrix should be readjusted in step 2 until it meets the rules of the consistency testing (CR < 0.1).
Step4: Calculate the weight of each indexes in the criteria layer and index layer.  Step 2: Construct the judgment matrix in each layer. The judgment matrix is constructed according to the Saaty-relative importance hierarchy table [17,21]. The Saaty-relative importance hierarchy table is shown in Table 1. Indictator i is slightly more important than indicator j 5 Indictator i is more important than indicator j 7 Indictator i is obviously more important than indicator j 9 Indictator i is absolutely more important than indicator j 2, 4, 6, 8 Between the hierarchy values mentioned above Step 3: Calculate the weight of the judgment matrix, and then check the consistency of the judgment matrix.
For checking the consistency, the compatibility index (CI), random consistency index (RI), and compatibility ratio (CR) proposed in [17] have been introduced.
where λ max denotes the maximum eigenvalue of the judgment matrix, n denotes the order of the judgment matrix, CI is the compatibility index, and RI is the random consistency index. According to [21], the relationship between n and RI is given as below: when n = 2, RI = 0; when n = 3, RI = 0.58; when n = 4, RI = 0.96.
The rules of consistency testing are as follows. If CR < 0.1, the judgment matrix can be accepted. If CR ≥ 0.1, the judgment matrix should be readjusted in step 2 until it meets the rules of the consistency testing (CR < 0.1).
Step 4: Calculate the weight of each indexes in the criteria layer and index layer.

Principle of SPA
Set pair analysis (SPA) [22][23][24] is a new method for processing the certainty and uncertainty problem. It has been widely used in mathematics, physics, and many other fields. The key idea of SPA is using the connection function to describe the identities, Electronics 2021, 10, 1274 4 of 12 differences and opposites of two sets (set A and set B) in a set pair. The connection function of set A and set B can be expressed as below.
where S, F, and P denote the number of the identities, the difference, and opposites, N is the total number of set pair features, Then, a, b, and c are called the identical degree, the difference degree, and the opposite degree between set A and set B.
In Equation (3), the connection function of the set pair can be seen and can describe the certainty and uncertainty. Where S and P are certainty, F is uncertainty. As specified in radar emitter recognition, as the electromagnetic environment is complex, the parameters of the emitter signals measured are always an uncertainty. Therefore, it is feasible to use SPA theory to evaluate the features of the emitter signals.

Feature Evaluation Based on SAHP
The SAHP algorithm, primarily founded on SPA and AHP, uses the AHP to get the weight vector of each layer. Then, SPA theory is introduced to solve the issue regarding heavy reliance on an expert's subjective experience in the feature comprehensive evaluation based on AHP. As mentioned above, SPA theory is adopted to build a new decision matrix to improve the objectivity of the evaluation. It is noted that in the following text, constructing the decision matrix is mainly based on the identity between the measurement values and the true values of different features, not on the difference and opposites of the set pair.
The steps of constructing the decision matrix based on SPA are given as below.
Step 1: Build the model of decision matrix D Suppose F is the target scheme set and X is the index set of the evaluated feature, then the mapping relationship between F and X is shown as below.
where d ij denotes the attribute mapping between the true value and the measured value on the ith feature of the jth index.
Step 2: Establish the evaluation scheme of decision matrix D Suppose T = (T 1 , T 2 , · · · , T n ) is the optimal solution of the feature to be evaluated. When the index in decision matrix D is considered as the profitability index, T i is defined as T i = max(d ij ). On the contrary, when the index in decision matrix D is considered as the cost index, T i is defined as T i = min(d ij ).
Step 3: Analyze the identity of decision matrix D.
The identical degree matrix ξ = (S ij ) m×n is constructed by computing the identical degree of the different indexes between the scheme set F and the optimal scheme set T. When the index is considered as the profitability index, S ij is defined as S ij = d ij /T i . While the index is considered as the cost index, S ij is defined as S ij = T i /d ij .
Step 4: Calculate the comprehensive evaluation results of the evaluated features.
According to the feature evaluation method by AHP, W = (ω 1 , ω 2 , · · · , ω m ) in the index layer can be obtained. Then, the comprehensive evaluation decision matrix E can be computed as below: where E is the comprehensive evaluation degree vector of all features to be evaluated.

Simulation
In this section, two simulations are presented to verify the validity and correctness of the proposed method. One uses SAHP to evaluate the comprehensive performance of each feature, compared with those by AHP. The second uses the evaluated features based on SAHP to check the effectiveness of the radar emitter recognition. In these two simulations, the features selected to be evaluated are the RF, PRI, PW, AOA, resemblance coefficient feature (RCF) in [25], and coherent feature (CF) in [26].

Weight Vector Calculation
From Figure 1 and Table 1, the judgment matrices (A, B1, B2, and B3) can be constructed as below: where a 12 = B1/B2 = 1/6 denotes the influence of complexity on the feature evaluation is clearly weaker than that of the accuracy. B 12 = C1/C2 = 3 denotes that the influence of time complexity is slightly more important on the feature evaluation than that of the spatial complexity. The meaning of other elements in matrices A, B1, B2, and B3 can be deduced in the same way. The maximum eigenvalue and the normalized eigenvector of the judgment matrix A can be computed as below.
For the judgment matrix A, the order is 3. According to Equations (1) and (2), CI A and CR A can be computed as below.
where CR A < 0.1, it denotes that the judgment matrix A can meet the rules of consistency testing. The weight of B1, B2, and B3 can be computed in the same way. The results are listed in Table 2.
In Table 2, the last column is the weights of each index in the index layer. The weights are W = (0.0639, 0.0213, 0.4746, 0.1107, 0.0623, 0.0902, 0.1804).
where m is total number of the indexes, w is the weight of ith index, k i is the score of ith index by the expert scoring standard, i = 1, 2, · · · , m. According to [17], the expert scoring standard is listed in Table 3. The comprehensive evaluation results of each feature based on AHP are listed in Table 4.  Table 4, it can be seen that the final score order of six features is CF > RCF > RF > PRI > AOA > PW.

Feature Evaluation Based on SAHP
As mentioned above, the difference between two algorithms is the construction method of the decision matrix. In the feature evaluation algorithm by AHP, the decision matrix is constructed according to the expert's scoring. While in the feature evaluation algorithm by SAHP, the decision matrix is constructed according to quantitative analysis between the true dates and the measured dates. Quantitative analyses [27] of each index by SAHP are given as follows.
(1) Time complexity (C1). The mean value of the feature extraction time in different signal-to-noise ratios (SNRs) is utilized to describe the time complexity. (2) Computational complexity (C2). This is measured by the resource consumption in the feature extraction.
(3) Recognition accuracy (C3). This is measured by the recognition accuracy of the feature in signal sorting.
(4) Separation among clusters (C4). This is measured by the probability of combined batches on the feature used for emitter signal sorting. The lower probability of combined batches for signal sorting, the higher separation of the clusters.
(5) Compactness within the cluster (C5). Considering the overlapping distribution of the signal features, the sum of the Euclidean distance among all samples in signal feature distribution is used to reflect the compactness within the cluster of the feature. The smaller the sum of Euclidean distance, the higher separation within the cluster.
(6) Noise immunity (C6) is calculated as follows. The parameters of the evaluated feature are measured across different SNRs (from 0 to 15 dB, every 3 dB) and the mean value of them is computed. Then, the Euclidean distance of all mean values are added to describe the jumping of the feature value across different SNRs, and the Euclidean distance is utilized to describe the performance of noise immunity.
(7) Stability (C7). This is the adaptability of the feature in different scenarios. It can be measured by the measurement error of different signal types in the same SNR.
Four types of signals are taken as the simulation experiment signals. There signals are continuous wave (CW) signal, linear frequency modulation (LFM) signal, binary phase shift keying (BPSK) signal, and quad-phase shift keying (QPSK) signal. The decision matrix is constructed as below.
The evaluation scheme of decision matrix is as follows. The recognition accuracy (C3) and the separation between clusters (C4) of the feature are defined as the profitability index, while the others are defined as the cost index. The target scheme set of feature evaluation can be computed as below.
The identical degree matrix can be computed as below.
According to Equation ( Simulations are carried out based on a dual core processor. The main frequency of the processor is 4 GHz and the memory is 16 GB. The evaluation results based on SAHP are  Table 5. Meanwhile, these are also compared with the evaluation results based on AHP. From Table 5, it can be seen that the evaluation results of each feature based on AHP and SAHP are different in numerical value. However, the ranks of each weight based on the two methods are the same. In the feature evaluation algorithm by SAHP, each element in the decision matrix is calculated according to the practical experimental dates. Thus, it is more reasonable and objective in the feature evaluation.

Emitter Recognition Based on the Evaluated Features
In order to check the effectiveness of the feature evaluation results based on SAHP, the convolutional neural network (CNN) classifier is taken for radar emitter recognition. Four features are selected respectively to form one-dimensional, two-dimensional, threedimensional, and four-dimensional feature vectors for the classifier. These features are RF, PRI, RCF, and CF. The structure diagram of CNN [28] in this experiment is shown in Figure 2.

 
According to Equation (8), the comprehensive evaluation decision matrix can be computed as below.
(0.8625, 0.7977, 0.7435, 0.7852, 0.8827, 0.9107) E  (15) Simulations are carried out based on a dual core processor. The main frequency of the processor is 4 GHz and the memory is 16 GB. The evaluation results based on SAHP are shown in Table 5. Meanwhile, these are also compared with the evaluation results based on AHP. From Table 5, it can be seen that the evaluation results of each feature based on AHP and SAHP are different in numerical value. However, the ranks of each weight based on the two methods are the same. In the feature evaluation algorithm by SAHP, each element in the decision matrix is calculated according to the practical experimental dates. Thus, it is more reasonable and objective in the feature evaluation.

Emitter Recognition Based on the Evaluated Features
In order to check the effectiveness of the feature evaluation results based on SAHP, the convolutional neural network (CNN) classifier is taken for radar emitter recognition. Four features are selected respectively to form one-dimensional, two-dimensional, three-dimensional, and four-dimensional feature vectors for the classifier. These features are RF, PRI, RCF, and CF. The structure diagram of CNN [28] in this experiment is shown in Figure 2.
. In this experiment, a CNN with an optimized 5C-2S-7C-2S network is employed according to [29]. The optimized parameters of the CNN are C1 = 5, S2 = 2, C3 = 7, and S4 = 2. The training parameters of the CNN are as below. The training is iterated five times, the training data is divided into three segments, the learning rate is 0.01, the weight decay is 0.0005, and the momentum constant is 0.8.

Performance Evaluation of Simulation 1
Simulation 1: Three radar emitters were utilized to check the recognition capability based on the evaluation results. The working mode of three emitters was the same: PRI staggering. All testing samples were taken from the emitter sample repository. Signal parameters are listed in Table 6. In Simulation 1, 240 samples were utilized for training and 120 samples were taken for testing. The recognition results by the CNN in different SNRs are shown in Table 7. As shown in Table 7, when a one-dimensional feature is used for emitter recognition, the recognition accuracy based on the CF feature is the highest. When the multidimensional features are utilized for recognition, it can be found that if the feature vector contains the CF feature, the recognition accuracy is obviously higher than the feature vector without the CF feature. That is, the CF feature has the higher recognition accuracy. Simulation results are consistent with the qualitative analysis results.

Performance Evaluation of Experiment 2
Simulation 2: Three radar emitters with different working modes were utilized to evaluate the recognition capability based on the evaluation results. The working mode of emitter1 is PRI staggering. The working mode of emitter2 is frequency agility. The working mode of emitter3 is PRI jittering. All testing samples were taken from the emitter sample repository. The signal parameters of three emitters are listed in Table 8. In simulation 2, 360 samples were utilized for training and 240 samples were taken for testing. The recognition results of the different feature vectors based on CNN in different SNRs are shown in Figure 3. In simulation 2, 360 samples were utilized for training and 240 samples were taken for testing. The recognition results of the different feature vectors based on CNN in different SNRs are shown in Figure 3. From Figure 3, we can observe that the recognition results are similar to the recognition results in Table 7. When the feature vector contains the CF feature, the recognition accuracy was significantly improved. Therefore, we can conclude that the proposed feature evaluation method by SAHP is effective in this paper. From Figure 3, we can observe that the recognition results are similar to the recognition results in Table 7. When the feature vector contains the CF feature, the recognition accuracy was significantly improved. Therefore, we can conclude that the proposed feature evaluation method by SAHP is effective in this paper.

Conclusions
This paper proposes a new feature evaluation method based on SAHP, which combines the SPA theory with AHP. The main idea of the proposed method is to use the SPA theory to establish the decision matrix. This can improve the objectivity of the decision matrix in comparison to the AHP algorithm. Simulations were carried out to verify the validity of the proposed feature evaluation method. Through the simulation results, it can be concluded that the proposed method can not only achieve the feature evaluation more objectively, but it can also provide a new idea for feature comparison and evaluation in radar emitter recognition. Nevertheless, the computation amount of the proposed method is larger than that of the evaluation algorithm by AHP. In further work, the model of constructing the decision matrix based on the SAHP will be optimized to reduce the calculation complexity. Meanwhile, many other features will be taken to be evaluated to verify the robustness of the proposed method.