Multi-Classifier Fusion for Open-Set Specific Emitter Identification

Abstract

To safeguard the privacy and security of IoT systems, specific emitter identification is utilized to recognize device identity with hardware characteristics. In view of the growing demand for identifying unknown devices, this paper aims to discuss open-set specific emitter identification. We firstly build up a problem formulation for open-set SEI by discussing the working mechanisms of radio signals and open-set recognition. And then it is pointed out that feature coincidence is an intractable challenge in open-set SEI. The reason, accounting for this phenome, is that pretrained fingerprint feature extractors are incapable of clustering unknown device features and differentiating them from known ones. Considering that feature coincidence leads to error recognition of unknown devices, we propose to fuse multi-classifiers in the decision layer to improve accuracy and recall. Three distinct inputs and four different fusion methods are adopted in this paper to implement multi-classifier fusion. The datasets collected at Huanghua Airport demonstrate that the proposed method can avoid the coincidence of feature space and achieve higher accuracy and recall.

1. Introduction

With the proliferation of IoT technologies and related products, privacy and security have been considerably exposed to the public. To prevent cyberattacks and deliberate deception, the IoT system is compelled to check device identity [1,2]. Key authentication, as the most common approach, faces the challenge of cracking passwords. This prospect of cracking passwords pushes specific emitter identification (SEI) to be imported into IoT security and privacy. Unlike key authentication, SEI extracts external fingerprint features and contrastively analyzes hardware characteristics, resulting in the recognition of device identities [3]. As a result of leveraging hardware uniqueness as an identification mark, SEI exhibits advantages in detecting impersonation attacks and relay attacks. Along with standalone applications, SEI can also be integrated with key authentication to provide a dual authentication system that ensures both privacy and security. Additionally, as a novel signal processing method in the field of remote sensing, SEI demonstrates prominent advantages in ship surveillance, air traffic control, and target tracking.

Currently, the majority of literature is centered on determining how to extract fingerprint features for use as classification marks, such as entropy [4], statistics [5], fractal [6], etc. Drawing a lesson from image processing, speech recognition, and defect detection [7], signals are fed into neural networks, containing CNN [4,8,9,10,11,12,13], LSTM [14], etc. Inputs of those neural networks can be various, such as time–frequency spectrum [4,9,11], bi-spectrum [8,10], and recurrence plots [11]. In particular, original in-phase and quadrature sampling points can be directly introduced into 1D-CNN networks [9,12,13]. Along with conventional network structures, novel neural networks also operate in certain situations. For example, adversarial network (AN) is used for adapting to frequency variation [15] and generative adversarial network (GAN) is utilized to mitigate the effect of intentional modulation [16,17]. These methods are used to recognize device identities in a certain feature space constructed by a corresponding feature extractor. Numerous researchers have sought to apply feature fusion to SEI in order to increase identification confidence [5,18,19,20]. The result reveals that feature fusion can contribute effectively to identification. The aforementioned methods can achieve excellent recognition accuracy under the premise of a close-set assumption. However, with the growing application of IoT devices, existing SEI systems fail to avoid facing unknown devices. Hence, it is essential to solve the open-set SEI problem, i.e., how to identify known devices and rule out unknown ones.

Open-set recognition (OSR) is utilized to deal with unknown situations that have not been learned by the models during training [21]. OSR has now strode forward widely in many fields, including face recognition, speech recognition, and object detection. Scholars have attempted to apply established OSR techniques to SEI, such as one-vs-all [22], OpenMax [22], and Siamese networks [23,24]. These methods are capable of identifying specific emitters under open-set assumptions. However, SEI confronts unique challenges. The aim of SEI is to recognize hardware diversity caused by process limitations [25]. Due to the unpredictability of hardware imperfection, predesigned systems face challenges in the description of unknown device characteristics. Hence, feature extractors are hard-to-cluster features of unknown devices, which may cause them to distribute randomly in the feature space or fall into known clusters. In addition, the lack of prior information about unknown devices may cause unknown clusters to partly overlap with known ones. The overlapped part brings challenges in distinguishing unknown clusters from known ones. Both these two cases result in a coincidence of features with known ones, preventing unknown devices from being rejected.

To address this issue, an open-set SEI method based on multi-classifier fusion is proposed in this paper. Multi-classifier fusion makes a joint decision via fusing identity declarations from multiple feature extractors, and in turn reduces recognition error of overlapped features. The proposed method contains three individual SEI channels and one classifier combiner. The inputs of those three channels are varied including in-phase and quadrature sampling points, frequency spectrum, and amplitude and phase. ResNet 18-layer (1d) acts as fingerprint feature extractors in those three channels. In the training process, each channel is separately trained with softmax as classifiers. When a test signal is input into the system, each channel makes an identity declaration with softmax-threshold or OpenMax as classifiers. Then, classifier combiners are used to fuse identity declarations and enhance identification performance.

The main contributions of this work are summarized as follows:

• On the basis of the open-set assumption, the proposed method extends SEI to IoT privacy and security. A mathematical model is developed for the open-set SEI, and it is illustrated that the intractable challenge is the feature coincidence resulting from subtle hardware differences. Additionally, it is proposed how to process the overlapped features using a classifier combiner framework.
• To avoid the coincidence of feature space, this paper proposes to adopt multi-classifier fusion to combine information from three feature spaces, including in-phase and quadrature sampling points, frequency spectrum, and amplitude and phase.
• Experiments on aircraft radars at Huanghua Airport demonstrate that the proposed method is capable of fusing information from various feature spaces effectively. The results indicate that this method enhances accuracy and recall and outperforms other methods.

The remainder of the paper is organized as follows. Section 2 develops a problem formulation for open-set SEI, introduces the related works in open-set SEI, and points out its intractable challenge. Aiming at the challenge of feature coincidence, we propose a method based on multi-classifier fusion in Section 3. Experiments in Section 4 demonstrate the high effectiveness of the proposed method. Finally, the paper is summarized and concluded in Section 5.

2. Background

This section begins by listing the definitions of notations in Section 2.1. The problem formulation is then presented in Section 2.2, comprising intercepted signal models and open-set SEI models. Section 2.3 reviews related literature on open-set SEI.

2.1. Notation and Definition

The notations used in this paper, along with their corresponding definitions, are listed in

Table 1. Definition of notation.

Notation	Definition	Notation	Definition
$x\left(t\right)$	Narrow-band radio signals	${ℝ}^m$	Feature space
$y\left(t\right)$	Signals with UIM	${ℝ}_{KKC}^m$	Subspace of feature space for KKC devices
$z\left(t\right)$	Intercepted signals	${ℝ}_{UUC}^m$	Subspace of feature space for UUC devices
$a\left(t\right)$	Amplitude modulation wave	$L\left(\cdot \right)$	Loss function
$f_o$	Carrier frequency	$h(\cdot )$	Classification space function
$\phi \left(t\right)$	Phase modulation wave	$\mathcal{H}$	Classification function space
$v\left(t\right)$	Additive noise	$R_e(h)$	Experimental risk
$UIM\left[\cdot \right]$	Function of UIM	$R_{\mathcal{O}}(h)$	Open-set risk
$\mathsf{\Delta }a\left(t\right)$	UIM of amplitude	$\mathsf{\lambda }$	Regularization constant.
$\mathsf{\Delta }f\left(t\right)$	UIM of frequency	$C_{KKC}$	The number of KKC devices
$\mathsf{\Delta }\phi \left(t\right)$	UIM of phase	$C_{UUC}$	The number of UUC devices
$T_s$	Sampling period	$\mathcal{O}$	Openness
$z\left(n\right)$	Sampled signals	${\mathcal{A}}_{\mathcal{O}}$	Accuracy for open-set SEI
$\Im \left(\cdot \right)$	Function of extracting features	${\mathcal{R}}_{\mathcal{O}}$	Recall for UUC devices
$fe{a}_i$	ith element in feature vector	$Inpu{t}_{IQ}$	Input matrix of in-phase and quadrature sampling points
$Fe{a}_k$	Feature vector for kth sample	$Inpu{t}_{Spec}$	Input matrix of frequency spectrum
$labe{l}_k$	Label for kth sample	$Inpu{t}_{AP}$	Input matrix of amplitude and phase
$predic{t}_k$	Predicted label for kth sample	$v$	The output vector of linear layer
$K$	Number of samples	$\widehat{v}$	Modified output vector
$N$	Length of signal	$\sigma \left(\cdot \right)$	Function of softmax layer
$m$	Dimension of feature space	$\epsilon$	Confidence threshold

.

2.2. Problem Formulation

This section introduces the core procedures of SEI by discussing the working mechanisms of radio signals.

As schematically illustrated in Figure 1, signals are generated in the transmitter and intercepted by receivers. Modulators and encoders generate baseband signals in the transmitter. Then, using radio-frequency mixers and power amplifiers, the signals are modulated to a certain radio frequency and amplitude. However, because of the limitations of processing techniques, imperfections in those hardware cause unintentional modulation, which provides theoretical support for device identification [25]. Receivers intercept signals via wireless channels and then use them for identity recognition. SEI performs three main phases in order to analyze UIM characteristics, extract fingerprint features, and recognize device identities: (1) preprocessing, (2) feature extraction, and (3) classification. Preprocessing enhances the signal quality via filtering noise, truncating signals, and sieving samples. To extract fingerprint features, conventional mathematical transformations and novel neural networks are used to map signals into one specific feature space. Then, classifiers divide the feature space and retrieve the identities of devices.

2.2.1. Model for Intercepted Signals

The narrow-band radio signals are denoted by

$$x\left(t\right) = a\left(t\right)\cos \left[2\pi f_{o}t + \phi \left(t\right)\right]\hspace{1em}0\le t\le T,$$

(1)

where $a\left(t\right)$ and $\phi \left(t\right)$ represent the modulation wave of amplitude and phase, respectively, $f_o$ is the carrier frequency, and $T$ is the lasting time. Considering the process limit, hardware imperfection unintentionally modulated signals are expressed as

$$y\left(t\right) = UIM\left[x\left(t\right)\right] = \left[a\left(t\right) + \mathsf{\Delta }a\left(t\right)\right]\cos \left[2\pi \left(f_o + \mathsf{\Delta }f\left(t\right)\right)t + \left(\phi \left(t\right) + \mathsf{\Delta }\phi \left(t\right)\right)\right],$$

(2)

where $UIM\left[\cdot \right]$ stands for unintentional modulation function. $\mathsf{\Delta }a\left(t\right)$, $\mathsf{\Delta }f\left(t\right)$, and $\mathsf{\Delta }\phi \left(t\right)$ denote unintentional modulation of amplitude, frequency, and phase, respectively. Due to the nuanced nature of fluctuation, it is assumed that $\left|\mathsf{\Delta }a\left(t\right)\right|\ll \left|a\left(t\right)\right|$, $\left|\mathsf{\Delta }f\left(t\right)\right|\ll \left|f\left(t\right)\right|$, and $\left|\mathsf{\Delta }\phi \left(t\right)\right|\ll \left|\phi \left(t\right)\right|$. During wireless transmission, signals are intercepted by receivers. In this research, wireless channels are assumed to introduce only additional noise, and the hardware of receivers is considered ideal without imperfection. As a result, the signal is

$$z\left(t\right) = \left[a\left(t\right) + \mathsf{\Delta }a\left(t\right)\right]\cos \left[2\pi \left(f_o + \mathsf{\Delta }f\left(t\right)\right)t + \left(\phi \left(t\right) + \mathsf{\Delta }\phi \left(t\right)\right)\right] + v\left(t\right).$$

(3)

Following sampling, received signals can be written as

$$z\left(n\right) = z(n{T}_s)\hspace{1em}1\le n\le N,$$

(4)

where $T_s$ denotes the uniform sampling time interval, and $N$ represents the length of a signal.

2.2.2. Model for Open-Set SEI

If the recognition problem is implemented by classification criteria, all devices can be classified into four categories [26], namely, known known class (KKC), known unknown class (KUC), unknown known class (UKC), and unknown unknown class (UUC). The first known or known represents whether or not training datasets are available. The second one indicates whether or not any additional information is included in the training. Typically, KUC serves as a background class while UKC is utilized for zero-shot learning. Hence, in this paper, datasets for open-set recognition are classified into two types: KKC and UUC. Assume the dataset contains $C$ devices, where $C_{KKC}$ ones can be used for training and testing, while $C_{UUC}$ ones for testing. Fingerprint feature extractors map signals $z\left(n\right)$ into one $m$-dimensional feature space ${ℝ}^m$. Then there is

$$Fea = \left[fe{a}_1 fe{a}_2\dots fe{a}_m\right] = \Im \left[z\left(n\right)\right],$$

(5)

where $\Im \left(\cdot \right)$ is the mapping function, $Fea$ represents a point in the feature space, and $fe{a}_i\left(i = 1, 2, \dots , m\right)$ is its ith element of feature space, as illustrated in Figure 2a. Let ${ℝ}_{KKC}^m$ and ${ℝ}_{UUC}^m$ be the known space and the unknown space, respectively. Then, there is ${ℝ}_{}^m = {ℝ}_{KKC}^m + {ℝ}_{UUC}^m$ and ${ℝ}_{KKC}^m\cap {ℝ}_{UUC}^m = 0$. For close-set recognition, assume that all test datasets are drawn from the known datasets. Classifiers divide the entire space into $C_{KKC}$ parts. When one feature falls within a certain subspace, the device identity can be inferred, as seen in Figure 2b. To design bounds appropriately, it is necessary to minimize the average loss, which is also referred to as experience risk:

$$\underset{h\in \mathcal{H}}{\arg \min }\left\{R_e(h)\right\},$$

(6)

$$R_e(h) = \frac{1}{K}{\displaystyle \sum _{k=1}^{K}L\left(labe{l}_k,h\left(Fe{a}_k\right)\right)},$$

(7)

where $K$ is the number of samples. $Fe{a}_k$ and $labe{l}_k$ are the feature and label of the kth sample, respectively. $L\left(\cdot \right)$ represents the loss function, such as cross-entropy loss, hinge loss, and center loss, and $h\left(\cdot \right)$ is the function that identifies the function space $\mathcal{H}$. Under such conditions, an SEI system would tag UUC devices as any KKC.

To address this issue, open-set recognition is introduced to SEI. Classifiers divide $C_{KKC}$ parts for KKC devices but leave the open space for UUC, as illustrated in Figure 2c. Additionally, the risk of the open space depends on the identification function and feature space. Qualitative description is as follows:

$$R_{\mathcal{O}}(h) = \frac{{\displaystyle {\mathsf{\int }}_{{ℝ}_{UUC}^m}h}(x)dx}{{\displaystyle {\mathsf{\int }}_{{ℝ}_{}^m}h}(x)dx}.$$

(8)

Hence, the problem can be written as follows:

$$\underset{h\in \mathcal{H}}{\arg \min }\left\{R_{\mathcal{O}}(h) + \mathsf{\lambda }{R}_e(h)\right\},$$

(9)

where $\mathsf{\lambda }$ is the regularization constant.

Hence, it can be deduced that the aim of SEI is to develop an appropriate fingerprint feature extracting mapping $\Im \left(\cdot \right)$ and classification space function $h\left(\cdot \right)$. The classification space function $h\left(\cdot \right)$ is critical in determining whether the SEI system can recognize unknown devices. Numerous mature classifiers and novel neural networks can perform well in the classification tasks under the close-set assumption. As a result, scholars concentrate more on how to design fingerprint feature extracting mapping $\Im \left(\cdot \right)$ by manual design or artificial intelligence. When it comes to open-set recognition, the classification space function $h\left(\cdot \right)$ seems to be more critical because it has a direct effect on the performance of recognizing unknown devices. Hence, this manuscript focuses on the design of the classification space function $h\left(\cdot \right)$, which is a critical distinction from close-set SEI.

2.3. Related Work

Only a little research has been undertaken on open-set SEI. The processes underlying open-set SEI are outlined in Figure 3. Methods of SEI in open-set recognition begin by developing an extractor of fingerprint features and a classifier using known device datasets. When testing the system, the same fingerprint feature extractor is used to extract features, followed by a different classifier to determine if the devices are known or unknown. To reject unknown devices, these classifiers first build decision boundaries based on the features of known devices learned during the training process. These boundaries can be built in a variety of ways, including spare representation, distance, and margin distribution. The signals are then evaluated to determine whether they are transmitted from an unknown device by determining if they fall within the decision boundaries. Methods in the open-set SEI can be classified into two categories: discriminative model-based (DM-based) and hybrid discriminative model-based (HDM-based), as illustrated in Figure 3.

Discriminative-model-based (DM-based) methods. In DM-based methods, feature extractors are designed in the same way as in the close-set problem, and discriminative classifiers are utilized for building decision boundaries. Luo [27] initially regarded OSR as a binary classification problem and used density-scaled classification margin two-class support vector data description (DSCM-TC-SVDD) to determine whether the emitter is an unknown. SEI incorporates more open-set recognition techniques as a result of the development of OSR. Hanna [22] compared discriminators, discriminating classifiers, one-vs-all, OpenMax, and autoencoders. The results indicated that when no background information is given and the number of known devices is less than five, one-vs-all and OpenMax outperform the other methods. However, a number of known devices for training, together with unknown devices for background information, are required to achieve an acceptable detection possibility using this method. In contrast to Hanna, Xu [28] attempted to build decision boundaries via modified intra-class splitting (ICS) using a transformer as a feature extractor. They score samples of known devices and split them into inners and outers while training a closed-set classification network. These outer samples can be utilized as unknown devices in conjunction with inner samples to train a new classification network where unknown devices are regarded as a new category. Although this method performs well at low openness, it performs poorly at excessive openness. Hence, Xu [29] proposed training such a new classification network using adversarial samples combined with outer samples. This approach is extremely performant in terms of image recognition, modulation recognition, and specific emitter identification. However, the performance of this approach is contingent upon the generation of adversarial samples. If the adversarial samples are unrelated to unknown devices, this method may potentially be less effective.

Hybrid-discriminative-model-based (HDM-based) methods. In the HDM-based method, feature extractors are modified for decreasing inter-class distance and increasing intra-class distance for accommodating the OSR task. Lin [30] proposed metric learning to facilitate feature aggregation for each known category, hence increasing the identification accuracy. He proposed a modified triplet loss and compared it to a center loss and triplet loss. By utilizing metric learning, it is possible to minimize inter-class distance, while maximizing intra-class distance. Learning from face recognition, Song [23] and Wu [24] incorporated Siamese networks into SEI. In Siamese networks, we determine whether it is an unknown device by computing the distance between two sub-networks. Although such methods can fulfill the OSR task in SEI, they still suffer from the problem of feature coincidence, which can result in errors in identifying unknown devices, especially with high openness.

These methods, adopted from other disciplines, can supplement the vacancy of open-set SEI. However, they unavoidably face a unique SEI challenge, i.e., the coincidence of features. Unlike other fields, the hardware diversity, resulting from manufacturing limitations, is unpredictable. Hence, it is hard for predesigned feature extractors to extract fingerprint features for unknown devices. Inappropriate feature extracting makes it hard to cluster unknown device features and differentiate them from known ones. Those two cases cause the coincidence of features, which results in the recognition error of unknown devices. In contrast to methods discussed previously, we propose to employ a classifier combiner in the decision layer to fuse decision boundaries across various feature spaces. By utilizing multi-classifier fusion, performance improvement is attained in detecting unknown devices and identifying known ones.

3. Methods

3.1. Framework of Classifier Confusion

This section proposes multi-classifier fusion as a technique to enhance identification performance by combining label predictions from various models. The framework of this method is depicted in Figure 4.

As illustrated in Figure 4, there exist three main channels in the framework. Each channel utilizes a residual neural network (ResNet) as a feature extractor to map signals into feature space. The inputs of networks are unique, containing in-phase and quadrature sampling point and the frequency spectrum, as well as the amplitude and phase. During the training process, features extracted from networks are fed into a close-set classifier, such as softmax. On the contrary, softmax-threshold and OpenMax are applied in testing experiments to detect UUC devices. Classifier combiners are used to fuse prediction results for achieving improved performance.

3.2. Input

Given the characteristics of unintentional modulation in the radio system, in-phase and quadrature sampling points and frequency spectrum, as well as amplitude and phase, are selected as network inputs. Assuming $z\left(n\right)$ as the signals, inputs can be calculated as follows:

• IQ sampling points

$$Inpu{t}_{IQ} = {\left[{\begin{array}{cc}z{\left(n\right)}^{\mathsf{T}}& \widehat{z}\left(n\right)\end{array}}^{\mathsf{T}}\right]}^{\mathsf{T}},$$

(10)

$$\widehat{z}\left(n\right) = \mathsf{H}\left[z\left(n\right)\right],$$

(11)

where $\mathsf{H}\left(\cdot \right)$ is the Hilbert transform, and ${\left(\cdot \right)}^{\mathsf{T}}$ is the transpose transform.

• Frequency spectrum

$$Inpu{t}_{Spec} = {\left[\begin{array}{cc}\mathrm{Re}{\left[Z\left(j\Omega \right)\right]}^{\mathsf{T}}& \mathrm{Im}{\left[Z\left(j\Omega \right)\right]}^{\mathsf{T}}\end{array}\right]}^{\mathsf{T}},$$

(12)

$$Z\left(j\Omega \right) = \mathrm{FFT}\left[z\left(n\right)\right],$$

(13)

where $\mathrm{FFT}\left(\cdot \right)$ is the Fourier transform.

• Amplitude and phase

$$Inpu{t}_{AP} = {\left[\begin{array}{cc}A{\left(n\right)}^{\mathsf{T}}& \phi {\left(n\right)}^{\mathsf{T}}\end{array}\right]}^{\mathsf{T}},$$

(14)

$$A\left(n\right) = \mathrm{abs}\left[z\left(n\right) + j\widehat{z}\left(n\right)\right]$$

(15)

When the signal has a length of $N$, each sample has a size of $2 \times N$. If $K$ is the number of samples, each network will have $K \times 2 \times N$ inputs.

3.3. Network

A residual neural network (ResNet) [31] is a type of artificial neural network utilizing skip connections to simplify the network, increase its computational efficiency, and avoid vanishing gradients. ResNet 18-layer(1d) was used as the backbone network in this paper, with a structure depicted in Figure 5.

Figure 5. Structure of the ResNet 18-layer(1d). (a) The basic block in the ResNet 18-layer. (b) A simplified structure of the ResNet 18-layer(1d).

Figure 5. Structure of the ResNet 18-layer(1d). (a) The basic block in the ResNet 18-layer. (b) A simplified structure of the ResNet 18-layer(1d).

As illustrated in Figure 5, the 18-layer ResNet is composed of eight basic blocks, each of which is implemented using double layer skips with nonlinearities (ReLU) and batch normalization.

3.4. Classifier

Softmax is employed for close-set recognition, whereas softmax-threshold and OpenMax are used for open-set recognition. The stacked basic blocks are followed by a linear layer in ResNet. They ensure that all signal features are mapped into a $C_{KKC}$-dimensional space ${ℝ}^{C_{KKC}}$. Assume that $v = \left(v_1 v_2\dots v_{C_{KKC}}\right)$ is the output of the linear layer acting as the input of classifiers. Then, the characteristics of the classifier will be as follows.

• Softmax

Softmax transfers $v$ into a probability distribution ${ℝ}^{C_{KKC}}\to {\left(0, 1\right)}^{C_{KKC}}$, expressed as

$$\sigma {\left(v\right)}_i = e^{v_i}/{\displaystyle \sum _{i=1}^{C_{KKC}}{e}^{v_i}}.$$

(16)

This is then followed by the calculation of the predicted label as

$$predict = \underset{i}{\arg \max }\left(\sigma \left(\widehat{v}\right)\right),$$

(17)

• Softmax-threshold

Softmax-threshold makes use of a confidence threshold $\epsilon$ to detect UUC devices. Considering $0$ as the label of UUC devices, the predicted labels are determined as

$$predict = \left\{\begin{array}{l}\underset{i}{\arg \max }\left(\sigma \left(\widehat{v}\right)\right)\hfill \\ 0\hfill \end{array}\hspace{1em}\begin{array}{l}if\max \left(\sigma \left(\widehat{v}\right)\right) > \epsilon \hfill \\ else\hfill \end{array}\right.,$$

(18)

• OpenMax

OpenMax estimates the distribution of the training dataset. If one feature deviates from the distribution of any KKC device, it can be classified as a UUC device. Distributions can be used to not only detect UUC devices, but also to revise the vector $v$.

Distributions are described in this section using extreme value theory (EVT). According to Ref. [32], the majority of maximum values follow the Gumbel distribution, defined as

$$F(x;\mu , \sigma , \xi ) = \left\{\begin{array}{ll}{e}^{-{(-y)}^{\alpha }}\hfill & y<0\hfill \\ 1\hfill & y\ge 0\hfill \end{array}\right.,$$

(19)

$$\xi = -{\alpha }^{-1}y = -(1 + \xi (x - \mu )/\sigma ),$$

(20)

where $\mu$ is the mean value of samples, and $\sigma$ and $\xi$ represent the shape and scale of the Gumbel distribution, respectively.

Then, we can revise $\alpha$ most primary elements of the vector $v$ as

$${\widehat{v}}_{s_j} = v_{s_j}{w}_{s_j}j = 1,2,\dots ,\alpha ,$$

(21)

$$w_{s_j} = 1-\frac{\alpha - j}{\alpha }\exp \left[-{\left(\frac{‖v - {\mu }_{s_j}‖}{{\sigma }_{s_j}}\right)}^{{\xi }_{s_j}}\right],$$

(22)

where $s = \left[s_1 s_2\dots s_{\alpha }\right] = \left[\arg \mathrm{sort}{\left(v\right)}_1 \arg \mathrm{sort}{\left(v\right)}_2\dots \arg \mathrm{sort}{\left(v\right)}_{\alpha }\right]$. To facilitate the identification of UUC devices, a new element is added to the revised vector, denoted by

$${\widehat{v}}_0 = {\displaystyle \sum _j^{\alpha }{v}_j\left(1 - w_j\right)}.$$

(23)

As with softmax, the vector $\widehat{v}$ is mapped into ${\left(0, 1\right)}^{C_{KKC} + 1}$. Each element in $\sigma \left(\widehat{v}\right)$ can be expressed as

$$\sigma {\left(\widehat{v}\right)}_i = e^{{\widehat{v}}_i}/{\displaystyle \sum _{i=0}^{C_{KKC}}{e}^{{\widehat{v}}_i}}.$$

(24)

The prediction is then calculated as

$$predict = \left\{\begin{array}{l}\underset{i}{\arg \max }\left(\sigma \left(\widehat{v}\right)\right)\hfill \\ 0\hfill \end{array}\hspace{1em}\begin{array}{l}if\max \left(\sigma \left(\widehat{v}\right)\right) > \epsilon \hfill \\ else\hfill \end{array}\right..$$

(25)

3.5. Combiner

According to the process level, information fusion is classified into three types: data layer information fusion, feature layer information fusion, and decision layer information fusion. Decision layer information fusion, also referred to as classifier combiner, is applied in this case. Additionally, classifier combiners are classified as either trainable or non-trainable. Since UUC datasets are hardly available, non-trainable ones are selected in this study. Details are provided in Figure 6.

Four representative classifiers are selected from this list, with three algebraic combiners and one voting-based method. Considering $M_{\mathrm{mod}el}$ as the number of classifiers, the fusion of those classifiers can be expressed as follows.

• Mean rule

Continuous-valued inputs $v$ of $M_{\mathrm{mod}el}$ classifiers are combined through a mean operation. Each element in the resulting vector represents an element of identification confidence. The label corresponding to the maximum value is regarded as the final label. If the classifier is softmax-threshold, the process can be expressed as

$$v = \frac{1}{M_{\mathrm{mod}el}}{\displaystyle \sum _{j=1}^{M_{\mathrm{mod}el}}\sigma \left(v^{\left\{j\right\}}\right)}$$

(26)

When it comes to OpenMax, functions are as

$$\widehat{v} = \frac{1}{M_{\mathrm{mod}el}}{\displaystyle \sum _{j=1}^{M_{\mathrm{mod}el}}\sigma \left({\widehat{v}}^{\left\{j\right\}}\right)}$$

(27)

For predicting the identities of devices, the vectors in (18) and (25) are replaced with (26) and (27), respectively.

• Maximum rule

In contrast to the mean rule, the combined vector’s elements are chosen based on the maximum value of $M_{\mathrm{mod}el}$ classifiers. The expressions for softmax-threshold and OpenMax are

$$v = \left[\underset{j}{\arg \max }\left(\sigma \left(v_1^{\left\{j\right\}}\right)\right) \underset{j}{\arg \max }\left(\sigma \left(v_2^{\left\{j\right\}}\right)\right)\dots \underset{j}{\arg \max }\left(\sigma \left(v_{C_{KKC}}^{\left\{j\right\}}\right)\right)\right]$$

(28)

$$\widehat{v} = \left[\underset{j}{\arg \max }\left(\sigma \left({\widehat{v}}_0^{\left\{j\right\}}\right)\right) \underset{j}{\arg \max }\left(\sigma \left({\widehat{v}}_1^{\left\{j\right\}}\right)\right)\dots \underset{j}{\arg \max }\left(\sigma \left({\widehat{v}}_{C_{KKC}}^{\left\{j\right\}}\right)\right)\right]$$

(29)

Similar to the mean rule, the vectors in (18) and (25) are replaced with (28) and (29) for predicting the identities of devices.

• Product rule

For the product rule, the vectors are combined via a product operation. Each element in the combined vector is the product of the corresponding elements in the $M_{\mathrm{mod}el}$ classifiers. The expressions for softmax-threshold and OpenMax are

$$v = \left[{\displaystyle \prod _j^{M_{\mathrm{mod}el}}\left(\sigma \left(v_1^{\left\{j\right\}}\right)\right)} {\displaystyle \prod _j^{M_{\mathrm{mod}el}}\left(\sigma \left(v_2^{\left\{j\right\}}\right)\right)}\dots {\displaystyle \prod _j^{M_{\mathrm{mod}el}}\left(\sigma \left(v_{C_{KKC}}^{\left\{j\right\}}\right)\right)}\right]$$

(30)

$$\widehat{v} = \left[{\displaystyle \prod _j^{M_{\mathrm{mod}el}}\left(\sigma \left({\widehat{v}}_0^{\left\{j\right\}}\right)\right)} {\displaystyle \prod _j^{M_{\mathrm{mod}el}}\left(\sigma \left({\widehat{v}}_1^{\left\{j\right\}}\right)\right)}\dots {\displaystyle \prod _j^{M_{\mathrm{mod}el}}\left(\sigma \left({\widehat{v}}_{C_{KKC}}^{\left\{j\right\}}\right)\right)}\right]$$

(31)

Similarly, the vectors in Equations (18) and (25) are replaced with (30) and (31) for predicting the identities of devices.

• Majority voting

In contrast to the three preceding combining rules, the majority voting is based solely on class labels. Classifiers predict the label for each sample separately, and the majority of those predicted labels are used as the final label, expressed as follows.

$$predict = \underset{j}{\mathrm{Majority}}\left[predic{t}^{\left\{j\right\}}\right]$$

(32)

3.6. Complexity Analysis

In this section, the complexity of the proposed method is discussed, containing ResNet 18-layer(1d) and classifier combiners. For ResNet 18-layer(1d), space complexity and time complexity are defined as

$$Time\sim O\left({\displaystyle \sum _{l=1}^D{M}_l\cdot K_l\cdot C_{l-1}\cdot C_l}\right)$$

(33)

$$Space\sim O\left({\displaystyle \sum _{l=1}^D{K}_l\cdot C_{l-1}\cdot C_l} + {\displaystyle \sum _{l=1}^D{M}_l\cdot C_l}\right)$$

(34)

where $O(\cdot )$ is the order of magnitude. In $l$th convolution layer, $C_{l-1}$ and $C_l$ are channels of input features and output features, respectively. $K_l$ and $M_l$, respectively, represent the size of convolution kernel and output feature. The item in (33) stands for the sum of floating-point operations (FLOPs). Two items in (34) are the sum of parameters (Params.) and feature map size (Fea. Map Size). Detailed analysis is in

Table 2. Complexity of ResNet 18-layer(1d).

	Layer Type	Kernel Size	Output Size	Time Complexity	Space Complexity
				FLOPs	Params.	Fea. Map Size
	Input Layer	$0$	$(N,2)$	$0$	$0$	$2N$
	Conv 1d	$(7,2,64)$	$(N/2,64)$	$448N$	$896$	$32N$
	MaxPool	$0$	$(N/4,64)$	$0$	$0$	$16N$
Conv_1	Conv 1d	$(3,64,64)$	$(N/4,64)$	$3072N$	$12,288$	$16N$
	Conv 1d	$(3,64,64)$	$(N/4,64)$	$3072N$	$12,288$	$16N$
	Conv 1d	$(3,64,64)$	$(N/4,64)$	$3072N$	$12,288$	$16N$
	Conv 1d	$(3,64,64)$	$(N/4,64)$	$3072N$	$12,288$	$16N$
Connection	Conv 1d	$(1,64,128)$	$(N/8,128)$	$1024N$	$4096$	$16N$
Conv_2	Conv 1d	$(3,64,128)$	$(N/8,128)$	$3072N$	$24,576$	$16N$
	Conv 1d	$(3,128,128)$	$(N/8,128)$	$6144N$	$41,952$	$16N$
	Conv 1d	$(3,128,128)$	$(N/8,128)$	$6144N$	$41,952$	$16N$
	Conv 1d	$(3,128,128)$	$(N/8,128)$	$6144N$	$41,952$	$16N$
Connection	Conv 1d	$(1,128,256)$	$(N/16,256)$	$2048N$	$32,768$	$16N$
Conv_3	Conv 1d	$(3,128,256)$	$(N/16,256)$	$6144N$	$98,304$	$16N$
	Conv 1d	$(3,256,256)$	$(N/16,256)$	$12,288N$	$196,608$	$16N$
	Conv 1d	$(3,256,256)$	$(N/16,256)$	$12,288N$	$196,608$	$16N$
	Conv 1d	$(3,256,256)$	$(N/16,256)$	$12,288N$	$196,608$	$16N$
Connection	Conv 1d	$(1,256,512)$	$(N/32,512)$	$4096N$	$131,072$	$16N$
Conv_4	Conv 1d	$(3,256,512)$	$(N/32,512)$	$12,288N$	$393,216$	$16N$
	Conv 1d	$(3,512,512)$	$(N/32,512)$	$24,576N$	$786,432$	$16N$
	Conv 1d	$(3,512,512)$	$(N/32,512)$	$24,576N$	$786,432$	$16N$
	Conv 1d	$(3,512,512)$	$(N/32,512)$	$24,576N$	$786,432$	$16N$
	AvgPool	$0$	$(1,512)$	$0$	$512$	$512$
	Full Connect	$(1,512,C_{KKC})$	$(1,C_{KKC})$	$512C_{KKC}$	$512C_{KKC}$	$C_{KKC}$
Totally				$0.65N+0.002C_{KKC}$ (MB)	$14+0.002C_{KKC}$(MB)	$354N+512+C_{KKC}$

.

From Table 2, it can be concluded that both time complexity and space complexity are $O(n)$. Now we consider the complexity of classifier combiners in the fusion method in the decision layer, as shown in

Table 3. Complexity of classifier combiners.

Classifier Combiner	Step	Time Complexity	Space Complexity
Algebraic Combiners	Input activate vector $v$ into classifier combiners.	0	$O(n)$
	Algebraic operations on activate vectors $v$, including maximum, mean, and product.	$O(n)$	$O(n)$
	Select the index of the maximum as the final predicted label.	$O(n)$	$O(n)$
Totally		$O(n)$	$O(n)$
Voting-based Method	Input activate vector $v$ into classifier combiners.	0	$O(n)$
	Calculate the number of time for each predicted label.	$O(n)$	$O(n)$
	Select the index of the maximum as the predicted label	$O(n)$	$O(n)$
Totally		$O(n)$	$O(n)$

.

As shown in Table 3, the time complexity and space complexity of non-trainable classifier combiners are $O(n)$. Summing the complexity of ResNet 18-layer(1d) and classifier combiners, time complexity and space complexity of the proposed method are $O(n)$. The use of ResNet 18-layer(1d) and non-trainable classifier combiners guarantee low time complexity and space complexity. Notice that using three neural networks for identification increases computation amount. For compensating this weakness, parallel computing is adopted during both the training and testing process.

4. Experiments

4.1. Experiment Environment

Experiments on civilian airplane radars were conducted at Huanghua Airport, China, as illustrated in Figure 7. Those radars from the same manufacturer are specially equipped on airplanes to report airplanes’ identities, altitudes, and other information to the airport, as well as to warn of impending air traffic. The carrier frequency of the collected signals is 1090 MHz, the sampling rate is 250 MHz, the pulse width is about 0.5 μs, and the signal-to-noise ratio is approximately 10 dB.

After signal truncation and sample sieving, a dataset of 20 devices is obtained. Each device environment is comprised of 240 samples, each of which contains 128 sampling points.

The software employed in this experiment includes Python 3.8.0, Pytorch 1.6.0, NVIDIA Cuda 10.2, and NVIDIA Cudnn 7.6.5. The model is run on a PC equipped with an Intel Xeon Gold 6230R CPU at 2.10 GHz, 128 GB of RAM, and an Nvidia Tesla V100S.

4.2. Evaluation Index

4.2.1. Openness

Openness quantifies the difficulty of open-set SEI tasks as a function of the number of KKC and UUC devices, expressed as

$$\mathcal{O} = 1-\sqrt{\frac{2 \times C_{TR}}{C_{TR} + C_{TE}}}$$

(35)

where $C_{TR}$ and $C_{TE}$ are the number of devices in the training and testing datasets, respectively. In most cases, testing datasets comprise devices in training datasets. Then, given $C_{TR} = C_{KKC}$, $C_{TE} = C_{KKC} + C_{UUC}$, (35) can be written as

$$\mathcal{O} = 1-\sqrt{\frac{2 \times C_{KKC}}{2 \times C_{KKC} + C_{UUC}}}$$

(36)

As a result of this, it can be deduced that openness is a monotonically increasing function of $C_{UUC}$, but a monotonically decreasing function of $C_{KKC}$. When there are more UUC devices but fewer KKC devices, the openness is larger, implying that the task of open-set SEI is more difficult. Specifically, when no UUC devices are given, openness equals zero, indicating that there are no difficult-to-identify UUC devices.

4.2.2. Accuracy

In the close-set SEI problem, accuracy is defined as

$$\mathcal{A} = \frac{{\displaystyle \sum _{i=1}^{C_{KKC}}\left(T{P}_i + T{N}_i\right)}}{{\displaystyle \sum _{i=1}^{C_{KKC}}\left(T{P}_i + T{N}_i + F{P}_i + F{N}_i\right)}}$$

(37)

When it comes to the open-set SEI problem, it is necessary to examine the rejection of UUC devices. After then, the accuracy is modified as

$${\mathcal{A}}_{\mathcal{O}} = \frac{{\displaystyle \sum _{i=1}^{C_{KKC}}\left(T{P}_i + T{N}_i\right)} + TU}{{\displaystyle \sum _{i=1}^{C_{KKC}}\left(T{P}_i + T{N}_i + F{P}_i + F{N}_i\right)} + \left(TU + FU\right)}$$

(38)

where $TU$ and $FU$ denote the rejection and acceptance of UUC devices, respectively. Accuracy describes the performance of the system in identifying KKC and UUC devices.

4.2.3. Recall

For information security purposes, it is more critical to recognize an unknown device than a known one. Hence, recall is also utilized here for unknown devices to demonstrate the capability of rejecting unknown devices.

$${\mathcal{R}}_{\mathcal{O}}=\frac{TU}{TU+FU}$$

(39)

The larger the recall, the better the system performs in rejecting unknown devices.

4.3. Plots of Features

In this experiment, the first ten devices are selected as KKC, whereas the last two act as UUC. Since the dimension of the vector $v$ is 10, the t-distributed stochastic neighbor embedding (t-TSNE) is taken to reduce the vector’s dimension for better visualization. The results are plotted in Figure 8.

Figure 8 depicts how features of UUC devices are distributed randomly in feature space. For example, UUC will be misclassified as KKC devices when features fall into ${ℝ}_{KKC}^m$. By comparing (a), (b), and (c), distinct inputs in different feature spaces can be inferred. Hence, performance can be improved by fusing information from different feature spaces. More specifically, it is rarely impossible for a UUC feature to fall into ${ℝ}_{KKC}^m$ in three conditions, which is beneficial to UUC detection accuracy. In addition, fusion can also contribute to the identification of KKC.

4.4. Efficiency

This section evaluates the efficiency of the proposed method. In this experiment designed in this section, the first ten devices are KKC, while the last ten are UUC. The confusion matrix of the proposed method, computed based on mean rule combiners, is depicted in Figure 9.

As illustrated in Figure 9, both methods are capable of detecting UUC devices while identifying the KKC devices. Softmax-threshold outperforms OpenMax in the KKC recognition, whereas OpenMax outperforms softmax-threshold in the UUC recognition. The most likely explanation is that OpenMax misclassifies KKC devices as UUC, lowering system accuracy.

4.5. Comparison with Other Open-Set SEI

For better evaluating the proposed method performance, three DM-based methods and two HDM-based methods are used for comparison, including softmax-threshold, OpenMax [22], intra-class splitting [28], metric learning based on triplet loss [30], and metric learning based on center loss [30]. The results are visualized in Figure 10.

In-phase and quadrature sampling points, frequency spectrum, and amplitude and phase act as inputs for those methods. Comparison experiments utilize one type of input each time, while the proposed method combines three of them. As shown in Figure 10a, the proposed method based on OpenMax can achieve higher accuracy than the other ones, while the proposed method based on softmax-threshold overperforms most of other methods. Additionally, it reveals that types of inputs for comparison methods have a subtle effect on accuracy. Figure 10b exposes that the proposed method based on OpenMax outperforms other methods in detecting unknown devices. This advantage of the proposed method can be highlighted with high openness, which will be discussed in the next section. Comparing three types of inputs, frequency spectrum is the least appropriate to act as inputs in rejecting unknown devices.

4.6. Effect of the Number of KKC and UUC Devices

This section addresses the effects of the number of UUC $C_{UUC}$ and KKC $C_{KKC}$ on the accuracy and recall. Note that accuracy encompasses both KKC and UUC, whereas the recall is solely concerned with UUC. In conclusion, accuracy implicates the capability to accurately identify a device, while recall refers to the ability to recognize UUC devices.

The effect of $C_{UUC}$ and $C_{KKC}$ on openness is depicted in Figure 11. Openness describes the degree of difficulty associated with open-set SEI. Recognizing more unknown devices and fewer known devices presents a greater challenge to the SEI system.

4.6.1. Effect of the Number of UUC Devices

Assuming that $C_{KKC} = 10$, the effect of $C_{UUC}$, which varies from 1 to 10, is analyzed in this experiment. The results are shown in Figure 12 and Figure 13.

Figure 12 and Figure 13 demonstrate how classifier fusion can improve accuracy and recall. The increment in recall is more significant than that of accuracy. When $C_{UUC}$ increases, accuracy declines proportionally, while the recall fiercely fluctuates. This is because $C_{UUC}$ affects the ability to recognize UUC devices directly. Softmax-based methods serve as a control experiment to demonstrate the limits of close-set SEI methods, as shown in Figure 12a and Figure 13a. It is of note that the recall of UUC in Figure 13a remains zero. This indicates that close-set SEI methods are incapable of identifying UUC devices, which is consistent with the findings in Section 3.4. Additionally, OpenMax-based methods outperform those based on softmax-threshold. Among these classifier fusion methods, those following the mean and product rules outperform the others.

4.6.2. Effect of the Number of KKC Devices

Assuming that $C_{UUC} = 10$, the effect of $C_{KKC}$, which varies from 1 to 10, is analyzed in this experiment. The results are shown in Figure 14 and Figure 15.

As illustrated in Figure 14 and Figure 15, the training process becomes more efficient when the increment of $C_{KKC}$ increases. Fewer features of UUC fall into ${ℝ}_{KKC}$, resulting in a higher accuracy and recall. Regardless of how much prior information training datasets provide, softmax restrictions preclude methods from identifying UUC devices, as seen in Figure 15a.

4.7. Combiner

To compare the effect of the combiner, this section compares various combinations of three channels, including each single channel, either two of the channels, or all three channels. Assuming $C_{UUC} = 15$ and $C_{KKC} = 5$, the comparison is conducted, and the results are displayed in Figure 16 and Figure 17.

As illustrated in Figure 16 and Figure 17, the confusion methods based on the mean rule and the product rule outperform the other two. The combination of Spec. and AP via the product rule and the combination of three via the mean rule are the optimal choices for boosting accuracy. When it comes to recall, the combination of two of them is preferable for the mean rule methods. Additionally, it is of note that the recall in Figure 17a remains zero. This is because softmax determines that methods are incapable of rejecting UUC devices, regardless of the decision layer combination.

5. Discussing

This paper can be regarded as the first trial in the field of applying fusion methods in open-set SEI. In future work, many improvements can be completed, such as fusing decisions given by different types of classifiers, extracting features via different network inputs, and applying trainable classifier combiners. In addition, aiming at the uniqueness challenge in open-set SEI, the coincidence of features, data layer information fusion, and feature layer information fusion can also be effective except for the decision layer fusion mentioned above.

6. Conclusions

To address the open-set recognition problem in the IoT security field, we developed a mathematical model and demonstrated that the unique challenge of open-set SEI is the coincidence of features. To solve this issue, a multi-classifier fusion method is proposed for open-set specific emitter identification. This method builds three fingerprint feature extractors for in-phase and quadrature sampling points, frequency spectrum, and amplitude and phase via ResNet. To overcome disadvantages of feature coincidence, classifier combiners are adopted to join identity declarations from those three feature extractors. Actual experiments conducted at Huanghua Airport demonstrate that the proposed method can enhance the accuracy of the identification system and the recall for unknown unknown class (UUC) devices.