Multi-Label Classification with Optimal Thresholding for Multi-Composition Spectroscopic Analysis

Abstract

In this paper, we implement multi-label neural networks with optimal thresholding to identify gas species among a multiple gas mixture in a cluttered environment. Using infrared absorption spectroscopy and tested on synthesized spectral datasets, our approach outperforms conventional binary relevance-partial least squares discriminant analysis when the signal-to-noise ratio and training sample size are sufficient.

1. Introduction

Spectroscopic analysis sees multiple applications in physics, chemistry, bioinformatics, geophysics, astronomy, etc. It has been widely used for detecting mineral samples [1], gas emission [2] and food volatiles [3]. Multivariate regression algorithms such as principal component regression [4] and partial least squares (PLS) [5] are fundamental and popular tools that have been successfully applied to spectroscopic analysis. Non-linear methods, such as support vector machine [6], genetic programming [7] and artificial neural networks (ANN) [1], are also adopted to increase prediction accuracy. These algorithms focus on either regression or single-label classification problems. Using multi-label classification to identify multiple chemical components from the spectrum is under explored. Unlike multi-classification counterparts that utilize multiple values of a single label to identify different spectroscopic components, multi-label methods adopt two or more output labels, one for each individual component. Consequently, relations between labels in multi-label tasks can be either independent or correlated.

The development of multi-label classification dates back to the 1990s when binary relevance (BR) [8] and the boosting method [9] were introduced to solve text categorization problems. A significant amount of research was done after that, and multi-label learning has been prosperous in areas such as natural language processing and image recognition [10,11,12,13,14,15,16]. Most of the multi-label classification algorithms fall into two basic categories: problem transformation and algorithm adaption. Problem transformation algorithms transform a multi-label problem into one or more single-label problems. After the transformation, existing single-label classifiers can be implemented to make predictions, and the combined outputs will be transformed back into multi-label representations. One of the simplest problem transformation methods is BR. It transforms a multi-label problem by splitting it into one binary problem for each label [17,18]. Under the assumption of label independence, it ignores the correlations between labels. If such an assumption fails, the label powerset (LP) and classifier chains (CC) are known transformation alternatives, where LP maps one subset of original labels into one class of the new single label [19] and CC passes label correlation information along a chain of classifiers [20]. For large label spaces where LP maps too many new labels to classify, label embedding techniques [21,22,23] divide the label space into subspaces and exploit the dependencies between them. In contrast, algorithm adaption methods modify existing single-label classifiers to produce multi-label outputs. For instance, the extensions of decision tree [24], AdaBoost [9], and k-nearest neighbours (KNN) [25,26] are all designed to deal with multi-label classification problems. The restricted Boltzmann machine [27], feedforward neural networks (FNN) [28,29], convolutional neural networks (CNN) [30,31], and recurrent neural networks (RNN) [32] are employed to characterize label dependency in image processing or to find feature representations in text classification. Those adaptive methods can identify multiple labels simultaneously and efficiently without being repeatedly trained for sets of labels or chains of classifiers.

Our application of multi-label learning for spectroscopic analysis adopts FNN with optimal thresholding (FNN-OT), which is an adaptive FNN model inspired by [28,29]. It will be compared with other problem transformation and algorithm adaption models that are extended from PLS and FNN. In this article, we will train all the models with simulated spectroscopic datasets and compare their results. It will be shown that for most evaluation metrics, the adaptive FNN model has the best performance.

2. Dataset

To synthesize the datasets, firstly, single gas molecule mid-infrared absorption cross-section spectra of ${\mathrm{C}}_2{\mathrm{H}}_6$, $\mathrm{C}{\mathrm{H}}_4$, $\mathrm{C}\mathrm{O}$, ${\mathrm{H}}_2\mathrm{O}$, $\mathrm{H}\mathrm{B}\mathrm{r}$, $\mathrm{H}\mathrm{C}\mathrm{l}$, $\mathrm{H}\mathrm{F}$, ${\mathrm{N}}_2\mathrm{O}$, and $\mathrm{N}\mathrm{O}$ measured at 296 K were selected from the high-resolution transmission molecular absorption (HITRAN) database [33] and are displayed in Figure 1a. These nine gases were selected to test the validity of our machine learning algorithm in many-gas environments. The gas spectra were down sampled to 1000 points equally spaced between 1 micrometer ($\mathsf{\mu }$m) and 7 $\mathsf{\mu }$m wavelengths. Secondly, the gas concentrations were randomly generated from a uniformly distributed probability density function such that the concentration of each gas was uniformly distributed between 0 and 10 micromolar ($\mathsf{\mu }$M). Thirdly, in real scenarios, gases could be partially correlated. To verify our model under partially correlated components, we introduced highly positive correlation between some gases so that their concentrations retained a pre-set correlation. The generation of uniformly distributed random variables with the target correlation matrix will be discussed in Appendix A. Further, in order to test the validity of our classification model, we modified the concentration matrix such that each gas only appeared in 50% of the gas mixture samples. Using the concentration matrix, the absorption spectrum of each gas mixture was synthesized using the Beer–Lambert law, assuming that the gas mixture was contained in a 10 cm long sensing region. Lastly, artificial Gaussian noises with a pre-set signal-to-noise ratio (SNR) were added to the constant light intensity across each wavelength point in order to obtain a closer-to-reality spectrum and evaluate our model’s accuracy in different noise environments. In our simulation, the math expression of the transmitted light intensity $I\left(\lambda \right)$ as a function of wavelength $\lambda$, given a light source with constant intensity $I_0$ and a Gaussian noise $\delta$, can be written as:

$$I\left(\lambda \right)=I_0(1+\delta )exp\{-\sum _i{N}_{i}b{\sigma }_i\left(\lambda \right)\}$$

(1)

where $N_i$ is the molecule concentration of the i-th gas in units of the number of particles per m${}^3$. b is the length of the sensing regime, and ${\sigma }_i\left(\lambda \right)$ is the corresponding absorption cross-section obtained from HITRAN. $\delta =N(0,{\sigma }_{\delta }^2)$ is a Gaussian random variable with zero mean and a variance of ${\sigma }_{\delta }^2$ such that the signal-to-noise ratio (SNR) of the light source in units of dB can be expressed as $\mathrm{SNR}\left(\mathrm{dB}\right)=-10{log}_{10}{\sigma }_{\delta }$. As an illustration, at SNR = $30\mathrm{dB}$, a sample gas mixture spectrum with each gas concentration of 7.024 $\mathsf{\mu }$M, 0 $\mathsf{\mu }$M, 8.908 $\mathsf{\mu }$M, 9.816 $\mathsf{\mu }$M, 7.793 $\mathsf{\mu }$M, 4.575 $\mathsf{\mu }$M, 0 $\mathsf{\mu }$M, 0 $\mathsf{\mu }$M and 1.987 $\mathsf{\mu }$M, respectively, is plotted in Figure 1b.

In this article, we used 12 datasets, and each had a pre-set SNR of $0\mathrm{dB}$, $10\mathrm{dB}$, $20\mathrm{dB}$, $30\mathrm{dB}$, $40\mathrm{dB}$ and $50\mathrm{dB}$. For each SNR, we generated two datasets respectively to represent uncorrelated and highly correlated cases. In uncorrelated cases, nine gas labels were mutually independent. In highly correlated cases, nine gases were evenly divided into three subsets. Gas labels within the same subset were highly correlated, and labels from different subsets were independent (Appendix A).

3. Algorithm

In single-label learning, a typical approach to classify an instance is to rank the probabilities (or scores) of all classes and choose the class with the highest probability as the prediction. For multi-label problems, the same ranking system can be used to compute scores for all labels instead, then a threshold will be determined to assign all labels whose scores are higher than the threshold to the sample. This label score-label prediction framework is the foundation of adapting NN for multi-label learning. In the FNN-OT model, scores of all labels need to be calculated for ranking purposes, and a threshold decision model will be employed to assign a set of labels to the sample in the label prediction step. The whole process of FNN-OT is shown in Figure 2a. Spectrum signals are firstly pre-processed by principal component analysis (PCA). The output principal components are the input features of an FNN model, which produces one output score for each gas. Output scores will be the input of a following optimal thresholding (OT). For every sample in the training set, its threshold will be determined by OT (details in Section 3.3). Then, the output scores and thresholds are the input and output variables of a new FNN model, which will be used to calculate thresholds for testing samples.

3.1. Feedforward Neural Networks

FNN has outstanding performance with large scale datasets [29]. As shown in Figure 2b, a typical FNN is formed by an input layer, an output layer and one or more hidden layers in-between. Each layer has a number of active neurons (circles without crosses in Figure 2b) that use the neuron outputs from previous layer as input and produce output to the neurons in the next layer. Previous research [29] showed that the single hidden layer model performs well on large scale text datasets. Furthermore, a preliminary test for the number of hidden layers was conducted on FNN without dropout. The results shown in Figure 1c indicate that FNN with only one hidden layer inside has the lowest Hamming loss, while the loss is higher in the cases of multiple hidden layers due to overfitting. Therefore, in our case of multi-label learning, a simple one hidden layer FNN model can achieve a state-of-the-art result with great computational efficiency. To get output score $\mathbf{s}$ based on input feature set $\mathbf{x}$, our FNN can be written as [34]:

$$\begin{array}{cc}\hfill \mathbf{h}& =f_h({\mathbf{W}}^{\left(1\right)}{\mathbf{P}}^{\left(1\right)}\mathbf{x}+{\mathbf{b}}^{\left(1\right)})\hfill \\ \hfill \mathbf{s}& =f_s({\mathbf{W}}^{\left(2\right)}{\mathbf{P}}^{\left(2\right)}\mathbf{h}+{\mathbf{b}}^{\left(2\right)})\hfill \end{array}$$

(2)

where $\mathbf{h}$ is a hidden layer that lies between the input and output layer, $f_h$ is the rectified linear units (ReLU) activation function in the hidden layer, $f_s$ is the sigmoid function for the output layer, and ${\mathbf{W}}^{\left(1\right)}$, ${\mathbf{W}}^{\left(2\right)}$, ${\mathbf{b}}^{\left(1\right)}$ and ${\mathbf{b}}^{\left(2\right)}$ are the parameters that need to be trained from data. In our model, the loss function $f_L(\mathbf{s},\mathbf{y})$ is defined as the cross-entropy of label score $\mathbf{s}$ and classification target $\mathbf{y}$, which can be expressed as:

$$f_L(\mathbf{s},\mathbf{y})=-\sum _{i=1}^L{y}_{i}log\left(s_i\right)+(1-y_i)log(1-s_i)$$

(3)

where L is the number of labels.

In our model, we adopted dropout to mitigate overfitting [34]. Dropout is a widely used method for preventing overfitting problems in neural networks. It randomly drops out a percentage of neurons in training, and the weights of remaining neurons will be trained by back-propagation [34]. Retention probability $\mathit{p}=(p_1,p_2)$ is the hyperparameter of dropout that will be tuned for our model. $p_1$ and $p_2$ are the probabilities of retaining units in the input and the hidden layer of the neural network model. Retention probabilities set for the FNN-OT model are the ones that result in minimum losses. The dropout is activated by two diagonal matrices of Bernoulli random variables ${\mathbf{P}}^{\left(1\right)}$ and ${\mathbf{P}}^{\left(2\right)}$ with parameters $p_1$ and $p_2$. Both parameters are retention probabilities of the input and hidden layer for dropout.

3.2. Principal Component Analysis

In both training and testing, the 1000-point absorbance spectra will be pre-processed with principal component analysis (PCA), and the principal components will be the input of the FNN model (x). PCA is a commonly used pre-processing method for spectroscopic datasets. It is conventionally employed to reduce the feature dimension by transferring original input variables into a smaller set of uncorrelated principal components (PC) that preserves the highest explained variance [35]. As shown in Figure 3a, at high SNR, PCA is an efficient technique for dimension reduction, as only a small number of PCs is sufficient to preserve most of the variances. However, when the SNR drops to below 30 dB, the variance of the original data is almost evenly projected into PCs. Under such circumstances, PCA will not be efficient for dimension reduction. Therefore, in a preliminary 10-fold test on the SNR = $40\mathrm{dB}$ dataset, Hamming loss has higher means when the number of PCs is less than the number of original inputs (blue line in Figure 3b). However, as shown in the same plot, when PCA is adopted in conjunction with dropout (blue markers), the Hamming loss is significantly reduced compared to the models that only adopt PCA (yellow markers), or dropout (red marker) or neither of them (purple marker). Therefore, in this article, we adopt PCA and dropout for all SNRs, not only for dimension reduction, but also for Hamming loss reductions.

3.3. Optimal Thresholding

Once we obtain the output score $\mathbf{s}$ for a specific instance, we need to find a threshold $t_i$ to convert the i-th label score $s_i$ in $\mathbf{s}$ to the i-th label predictions ${\widehat{y}}_i$ in $\widehat{\mathbf{y}}$. Here, ${\widehat{y}}_i$ can be expressed by an indicator function ${\widehat{y}}_i=1(s_i>t_i)$. That is, for the i-th specific gas component label that has a score higher than $t_i$, the prediction is 1 and 0 otherwise, representing the existence/non-existence of that gas component in the spectrum.

For binary classification problems in single-label learning, the sigmoid activation function of the output layer results in output scores that are between 0 and 1, and those output scores are often interpreted as the probabilities of the two possible classes. For each sample in the testing set, its predicted class will be the one with more than 0.5 probability (output score), so the classifier can be viewed as an FNN model with a threshold $t=0.5$. As shown in our Results Section, mislabelling of an extremely low concentration of a specific gas species as absent from the sample occurs more frequently than mislabelling a non-existing gas species as existing in the sample. This results in an imbalance between recall and precision. To re-balance recall and precision for higher ${\mathrm{F}}_1$, adopting an optimal threshold t for each label in each instance is desirable. For samples in the training set, the method of determining t is illustrated in Figure 4a. Suppose we have obtained output scores for all nine labels of a gas mixture. Three of them (blue ones) have the ground truth value of 1 (gas species exists in the sample), and the rest of the labels in red are 0 (gas species is absent in the sample). Then, we calculate the ${\mathrm{F}}_1$ scores for the three candidates $t_1,t_2$ and $t_3$ of t (dash lines), and the candidate with the highest ${\mathrm{F}}_1$ score, which is $t_2$ in this example, is the t we need. In our model, we use output scores to calculate the candidates of t. For each sample, nine output scores will be formed into an increasing order: $s_1\le s_2\le ,\dots ,\le s_9$. Since the sigmoid function is used in the output layer, all output scores are between 0 and 1. Therefore, the ten threshold candidates will be:

$$0,\frac{s_1+s_2}{2},\frac{s_2+s_3}{2},\dots ,\frac{s_8+s_9}{2},1$$

In order to get thresholds for all instances in the testing set systematically, we assume that threshold t is determined by the label scores $\mathbf{s}$, and their relationship can be recognized by the following FNN model:

$$\begin{array}{cc}\hfill {\mathbf{h}}_t& =f_h({\mathbf{W}}_t^{\left(1\right)}\mathbf{s}+{\mathbf{b}}_t^{\left(1\right)})\hfill \\ \hfill \widehat{t}& ={\mathbf{W}}_t^{\left(2\right)}{\mathbf{h}}_t+{\mathbf{b}}_t^{\left(2\right)}\hfill \end{array}$$

(4)

where ${\mathbf{h}}_t$ is a hidden layer with ReLU activation function $f_h$ and ${\mathbf{W}}_t^{\left(1\right)}$, ${\mathbf{W}}_t^{\left(2\right)}$, ${\mathbf{b}}_t^{\left(1\right)}$ and ${\mathbf{b}}_t^{\left(2\right)}$ are the parameters that need to be estimated. We will use instances in the training set to train the FNN model, and the loss function is the mean squared error between t and $\widehat{t}$.

3.4. Evaluation Metrics

Hamming loss is a common evaluation metric for multi-label classification tasks. It is the fraction of the wrong predictions to the number of total gas labels of all testing samples. Mathematically, it is defined as:

$$Hammingloss=\frac{1}{L·N}\sum _{i=1}^N\sum _{j=1}^L\frac{F{N}_{ij}+F{P}_{ij}}{T{N}_{ij}+T{P}_{ij}+F{N}_{ij}+F{P}_{ij}}$$

(5)

where $T{N}_{ij},T{P}_{ij},F{N}_{ij}$ and $F{P}_{ij}$ are true negatives, true positives, false negatives and false positives of the $i\mathrm{th}$ class of the $j\mathrm{th}$ label.

To evaluate our models, we also used micro-averaged recall, precision and ${\mathrm{F}}_1$ as our figures of merit [10]. In our context, a true negative (TN) is the absence of a certain gas that has been correctly predicted in a sample. Similarly, a true positive (TP) is the case that an existing gas is marked as present in a sample. A false negative (FN) is the case that the classifier fails to identify an existing gas, and a false positive (FP) is a false alarm where the classifier identifies a non-existent gas.

4. Results and Discussions

4.1. Hyper-Parameter Tuning

In our research, we used TensorFlow to implement our FNN-OT and Adam as our optimizer. In the first step, we tuned hyper-parameters such as the dropout rate and training sample size of the FNN-OT model with the SNR = $30\mathrm{dB}$ dataset.

4.1.1. Dropout

In order to tune the hyper-parameters for dropout, a grid search was conducted on retention probabilities $\mathit{p}=(p_1,p_2)$ of the input and hidden layers. A typical choice of retention rate is 0.8 for the input layer and 0.5 for the hidden layer [34], and a preliminary search on our datasets showed that the optimal choice of $\mathit{p}$ was around (0.95, 0.2). More detailed tests on $p_1$ and $p_2$ were done around this point, and the results of all SNRs are shown in Figure 5. Combinations of $p_1$ and $p_2$ with the lowest Hamming loss will be applied to our model.

4.1.2. Training Sample Size

To determine the number of training samples that are sufficient for our models, we plotted the learning curve as shown in Figure 4b. Here, FNN-OT is compared with PLS-BR (Appendix B) and FNN with a 0.5 threshold.

As shown in the learning curves plot, we changed the number of samples in the training set while keeping the 20,000-sample testing set intact. Both training (solid lines) and testing (dashed lines) Hamming losses are plotted as a function of training samples. At an SNR of $30\mathrm{dB}$, without dropout (blue markers), our FNN-OT model displayed large variance and low bias as the training loss was almost zero, while the testing loss was above 0.1 even when the training sample size was around 100,000. This is a clear indication of overfitting for training samples fewer than 100,000. In contrast, by adopting dropout (red markers), the overfitting issue was solved, and both training and testing loss converged to around 0.05 at around 100,000 training samples. In comparison, PLS-BR (green markers) did not display overfitting at the aforementioned sample size. However, the converged training and testing losses were higher (>0.1) than our FNN-OT model with dropout, indicating that our model outperformed this conventional technique. Nevertheless, the plot clearly showed that it was sufficient to use around 100,000 samples to train our FNN-OT with the dropout model.

4.2. Performance Comparison of Mutually Independent Gas Data

Parameters of PCA and FNN models will be trained in the 80,000-sample training sets and deployed in the 20,000-sample test sets.

We first compared our model using the datasets where all gas components were mutually independent. Figure 6 and

Table 1. Numerical results of Figure 6a–c.

	Micro-Averaged Precision
	FNN-OT	FNN	PLS-BR
$0\mathrm{dB}$	0.50 ± 0.03	0.499 ± 0.001	0.4998 ± 0.0004
$10\mathrm{dB}$	0.562 ± 0.002	0.545 ± 0.003	0.5561 ± 0.0004
$20\mathrm{dB}$	0.81 ± 0.03	0.83 ± 0.02	0.8124 ± 0.0003
$30\mathrm{dB}$	0.983 ± 0.002	0.985 ± 0.006	0.9935 ± 0.0001
$40\mathrm{dB}$	0.9992 ± 0.0002	0.9996 ± 0.0001	1 ± 0
$50\mathrm{dB}$	0.998 ± 0.001	0.998 ± 0.002	1 ± 0
	Micro-Averaged Recall
	FNN-OT	FNN	PLS-BR
$0\mathrm{dB}$	0.54 ± 0.03	0.502 ± 0.006	0.504 ± 0.001
$10\mathrm{dB}$	0.594 ± 0.005	0.539 ± 0.003	0.561 ± 0.001
$20\mathrm{dB}$	0.78 ± 0.01	0.71 ± 0.03	0.7083 ± 0.0003
$30\mathrm{dB}$	0.908 ± 0.002	0.89 ± 0.02	0.7488 ± 0.0002
$40\mathrm{dB}$	0.953 ± 0.002	0.942 ± 0.004	0.7492 ± 0.0003
$50\mathrm{dB}$	0.965 ± 0.003	0.96 ± 0.02	0.7507 ± 0.0002
	Micro-Averaged${\mathbf{F}}_{\mathbf{1}}$Score
	FNN-OT	FNN	PLS-BR
$0\mathrm{dB}$	0.52 ± 0.02	0.501 ± 0.003	0.5017 ± 0.0007
$10\mathrm{dB}$	0.578 ± 0.003	0.542 ± 0.003	0.5587 ± 0.0008
$20\mathrm{dB}$	0.79 ± 0.02	0.764 ± 0.008	0.7568 ± 0.0003
$30\mathrm{dB}$	0.9439 ± 0.0007	0.94 ± 0.01	0.8539 ± 0.0002
$40\mathrm{dB}$	0.9755 ± 0.0009	0.970 ± 0.002	0.8566 ± 0.0002
$50\mathrm{dB}$	0.981 ± 0.002	0.978 ± 0.009	0.8576 ± 0.0001

present the micro-averaged precision, recall and ${\mathrm{F}}_1$ score at six different SNRs. As expected, all models performed better at higher SNRs. When SNR was $0\mathrm{dB}$, all three classifiers failed to identify gases because a 0.5 micro-${\mathrm{F}}_1$ score is as good as a random guess. Across all SNRs, FNN-OT yielded better precision, recall and ${\mathrm{F}}_1$ than the conventional PLS-BR, clearly indicating it as a superior approach for gas identification.

FNN-OT also outperformed FNN in most cases at the cost of slightly longer computing time. It took FNN-OT around 640 s to complete one of the 10 folds, and for FNN, the computing time was about 430 s. In some cases, when the SNR was $0\mathrm{dB}$, FNN-OT had slightly lower precision compared to FNN. This resulted from the optimal thresholding system where thresholds were the ones that could maximize the ${\mathrm{F}}_1$ scores. FNN-OT scarified precision to reach a higher overall ${\mathrm{F}}_1$ score. Figure 6 also illustrates that all three models displayed higher values of precision than recall. This was due to the fact that most mislabelling occurred when a gas species’ concentration was too low to produce a detectable signal above the noise background, and all model would mistakenly predict the absence of that gas and produce an FN. However, as is evident from Figure 6b, selecting the optimal threshold would significantly reduce the occurrence of FN and increase recall without significantly reducing the precision, resulting in a better ${\mathrm{F}}_1$ score. This clearly justified the necessity of adopting FNN-OT.

The advantages of FNN-OT were further confirmed by comparing the minimum detectable concentrations of nine gases as in Figure 7. As shown, both FNN-OT and FNN consistently showed lower minimum detectable concentration at all SNRs, while in general, FNN-OT outperformed FNN.

4.3. Performance Comparison for Highly Correlated Gas Data

We further applied our models to the cases when the gases were correlated. As shown in Figure 8a–c and

Table 2. Micro-averaged metrics of highly correlated gas data at different SNRs.

	Micro-Averaged Precision
	FNN-OT	FNN	PLS-BR
$0\mathrm{dB}$	0.70 ± 0.08	0.4979 ± 0.0008	0.498 ± 0.001
$10\mathrm{dB}$	0.71 ± 0.02	0.594 ± 0.006	0.6030 ± 0.0005
$20\mathrm{dB}$	0.89 ± 0.01	0.896 ± 0.003	0.9114 ± 0.0003
$30\mathrm{dB}$	0.987 ± 0.001	0.987 ± 0.001	0.9961 ± 0.0001
$40\mathrm{dB}$	0.9989 ± 0.0003	0.9991 ± 0.0002	1 ± 0
$50\mathrm{dB}$	0.9996 ± 0.0005	0.9998 ± 0.0001	0.9926 ± 0.0007
	Micro-Averaged Recall
	FNN-OT	FNN	PLS-BR
$0\mathrm{dB}$	0.6 ± 0.1	0.492 ± 0.008	0.495 ± 0.002
$10\mathrm{dB}$	0.64 ± 0.07	0.573 ± 0.007	0.599 ± 0.001
$20\mathrm{dB}$	0.81 ± 0.01	0.777 ± 0.005	0.7232 ± 0.0004
$30\mathrm{dB}$	0.9388 ± 0.0007	0.934 ± 0.002	0.7497 ± 0.0003
$40\mathrm{dB}$	0.960 ± 0.002	0.957 ± 0.001	0.7505 ± 0.0002
$50\mathrm{dB}$	0.9692 ± 0.0007	0.965 ± 0.004	0.7497 ± 0.0002
	Micro-Averaged${\mathbf{F}}_{\mathbf{1}}$Score
	FNN-OT	FNN	PLS-BR
$0\mathrm{dB}$	0.65 ± 0.07	0.495 ± 0.004	0.496 ± 0.001
$10\mathrm{dB}$	0.67 ± 0.03	0.583 ± 0.007	0.6011 ± 0.0007
$20\mathrm{dB}$	0.846 ± 0.003	0.832 ± 0.002	0.8064 ± 0.0002
$30\mathrm{dB}$	0.9625 ± 0.0004	0.9597 ± 0.0006	0.8555 ± 0.0002
$40\mathrm{dB}$	0.9791 ± 0.0009	0.9776 ± 0.0006	0.8575 ± 0.0002
$50\mathrm{dB}$	0.9841 ± 0.0004	0.9821 ± 0.0020	0.8542 ± 0.0004

, when SNR was above $20\mathrm{dB}$, the performance of the three models was similar to the uncorrelated case, and FNN-OT outperformed. Further, at SNR = $0\mathrm{dB}$ or $10\mathrm{dB}$, FNN-OT significantly outperformed the other two models and its own results of the uncorrelated case due to the fact that FNN-OT could collaboratively identify gas species by organizing their correlation, while FNN and PLS-BR were not capable of this. Since PLS-BR ignored label correlations completely, in Figure 8d and

Table 3. Micro-averaged metrics and computing time of highly correlated gas data at SNR = $30\mathrm{dB}$. FNN-OT and FNN were trained on a GPU. Other models were trained on a CPU.

	Precision	Recall	${\mathbf{F}}_1$ Score	Computing Time
FNN-OT	0.987 ± 0.001	0.9388 ± 0.0007	0.9625 ± 0.0004	640 s
FNN	0.987 ± 0.001	0.934 ± 0.002	0.9597 ± 0.0006	440 s
PLS-BR	0.9961 ± 0.0001	0.7497 ± 0.0003	0.8555 ± 0.0002	100 s
SVM-CC	0.9855 ± 0.0007	0.8365 ± 0.0009	0.9049 ± 0.0004	1440 s
CLEMS	0.819 ± 0.002	0.738 ± 0.002	0.7765 ± 0.0008	12,000 s
RF-LP	0.691 ± 0.002	0.650 ± 0.004	0.670 ± 0.002	720 s

, three models, support vector machine-classifier chains (SVM-CC), cost-sensitive label embedding with multidimensional scaling (CLEMS) and random forest-label powerset (RF-LP), were added as additional base models for comparison. SVM-CC is the combination of a common classifier support vector machine (SVM) and a problem transformation method classifier chains (CC), where CC exploits label correlation information with a chain of classifiers. CLEMS is a well performing label embedding method that transforms original labels into new embedded space. Along with CLEMS, we applied the random forest regressor and the multi-label k-nearest neighbor (MLKNN) classifier, which is a common combination used for label embedding. RF-LP uses the label powerset to map the combinations of original labels into new single labels for random forest to classify. The results in Figure 8d show that SVM-CC, CLEMS and RF-LP had lower micro-averaged precision, recall and ${\mathrm{F}}_1$ score compared to FNN-OT.

All models were trained and tested on a home assembled desktop computer that had 64 GB DDR4 memory, a Micro-Star Z370 Gaming Plus motherboard and a 3.20 GHz Intel i7-8700 CPU with 12 units of processors. It had two graphics processing units (GPUs) installed: one was an NVidia GeForce GTX 1070 with 1920 CUDA cores and 8 GB GDDR5 memory, and the other was an NVidia GeForce GTX 1050 with 768 CUDA cores and 4 GB GDDR5 memory. Here, both FNN and FNN-OT were implemented using Google’s TensorFlow and trained on the GTX 1070 GPU. Other models were implemented using skicit-learn and skicit-multilearn packages and trained on a CPU. During training of FNN and FNN-OT, the typical volatile GPU utilization was around 64%, and the typical CPU load was around 5.0 when training other models. The computing time of one of the 10-fold cross-validations for all six models is compared in the last column of Table 3. With the assistance of the GPU, FNN-OT outperformed SVM-CC, CLEMS and RF-LP in computing time. On the other hand, PLS-BR had the shortest computing time of 100 s. The computing time of CLEMS was the longest (about 3.5 h) among all models.

5. Conclusions

In conclusion, by selecting optimal thresholds, FNN-OT outperformed conventional PLS-BR, SVM-CC, CLEMS and FNN in two aspects. FNN-OT could dynamically select a threshold to reduce FN events. In addition, FNN-OT was capable of utilizing the correlation among the components to enhance its classification capability. Both of these unique features make FNN-OT a favourable choice for spectroscopic analysis in cluttered environments.