^{1}

^{2}

^{3}

^{*}

^{2}

^{2}

^{2}

^{2}

^{2}

^{4}

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

When extracting discriminative features from multimodal data, current methods rarely concern themselves with the data distribution. In this paper, we present an assumption that is consistent with the viewpoint of discrimination, that is, a person's overall biometric data should be regarded as one class in the input space, and his different biometric data can form different Gaussians distributions,

Multimodal biometric recognition techniques use multi-source features together in order to obtain integrated information to obtain more essential data about the same object. This is an active research direction in the biometric community, for it could overcome many problems that bother traditional single-modal biometric system, such as the instability in one's feature extraction, noisy sensor data, restricted degree of freedom, and unacceptable error rates. Information fusion is usually conducted on three levels,

Linear discriminant analysis (LDA) is a popular and widely used supervised discriminant analysis method [

In recent years, many kernel discriminant methods have been presented to extract nonlinear discriminative features and enhance the classification performance of linear discrimination techniques, such as kernel discriminant analysis (KDA) [

In this paper, we have developed a novel multimodal feature extraction and recognition approach based on linear and nonlinear discriminant analysis technique. We adopt the feature fusion strategy, as features play a critical role in multimodal biometric recognition. More specifically, we try to answer the question of how to effectively obtain discriminative features from multimodal biometric data. Some related works have appeared in the literature. In [

While current methods generally extract discriminative features from multimodal data technically, they have rarely considered the data distribution. In this paper, we present an assumption that is consistent with the viewpoint of discrimination, that is, in the same feature space, one person's different biometric identifier data can form different Gaussians, and thus his overall biometric data can be described using mixture-Gaussian models. Although LDA has been widely used in biometrics to extract discriminative features, it has the limits that it can only handle the data of one person that forms a single Gaussian distribution. However, as we pointed out above, in multimodal analysis, different biometric identifier data of one person can form mixture-Gaussians. Fortunately, subclass discriminant analysis (SDA) [

Based on the analysis above, in this paper we propose a novel multimodal biometric data feature extraction scheme based on subclass discriminant analysis (SDA) [

Two solutions are presented to solve the small sample size problem encountered in calculating the optimal transform. One is to initially do PCA preprocessing, and the other is to employ the generalized singular value decomposition (GSVD) [^{n}

We evaluate the proposed approaches on two face databases (AR and FRGC), and the PolyU palmprint database, and compare the results with related methods that also tend to extract descriptive features from multimodal data. Experimental results show that our approaches achieve higher recognition rates than compared methods, and also get better verification performance than compared methods. It is worthwhile to point out that, although the proposed approaches are validated on data of two modalities, it could be easily extended to multimodal biometric data recognition.

The rest of this paper is organized as follows: Section 2 describes the related work. Section 3 presents our approach. In Section 4, we present the kernelization of our approach. Experiments and results are given in Section 5 and conclusions are drawn in Section 6.

In this section, we first briefly introduce some typical multimodal biometrics fusion techniques such as pixel level fusion [

The general idea of pixel level fusion [

In [_{i}, y_{i}^{th}_{i}, i.e.

On the other hand, the parallel fusion strategy combines the features into a complex vector _{i}, i.e.

Yang _{i}_{i}

Subclass discriminant analysis (SDA) [_{B}_{W}_{i}_{ij}= n_{ij}/n^{th}_{ij}^{th}_{W}^{−1}Σ_{B}

Kernel subclass discriminant analysis (KSDA) is the nonlinear extension of SDA based on kernel functions [_{ij}^{th}^{th}

In kernel PCA [_{i}

Like PCA, the eigenvalue equation _{1}),… Ø (_{M}_{i}_{1},…_{M}_{1},…_{M}_{1},…_{M}, K_{i}

In this section, we propose a novel multimodal biometric feature extraction scheme based on SDA. Two solutions are separately introduced to avoid the singular problem in SDA, which are PCA and GSVD. Then we present the algorithm procedures of the proposed SDA-PCA and SDA-GSVD approaches.

For simplicity, we take two typical types of biometric data as examples in this paper. One is the face data, and the other is the palmprint data. From the viewpoint of discrimination, it is quite natural to assume that the overall biometric data one person may be regarded as one class. Moreover, his palmprint and face data can be regarded as two subclasses of this class in the same feature space. An example of two person's face and palmprint samples is shown in

As can be seen from

Let
^{th}_{c}_{B}_{W}_{c}, _{ij}_{kl}_{c}/N

Let be the optimal transform vector to be calculated, and then it can be obtained by:

The within-class matrix _{W}

The first solution is to first apply PCA to project each image
_{SDA}, i.e._{W}^{−1}_{B}

Based on Formula (14), the rank of _{W}

Let
_{SDA}

After the optimal transformations _{1} and _{2} are obtained, we project the face sample

Then, features derived from face and palmprint are fused used using serial fusion strategy and used for classification:

While PCA is a popular way to overcome the singular problem and accelerate computation, it may cause information loss. Therefore, we present a second way to overcome the singularity problem by employing GSVD. First, we rewrite the between-class scatter matrix and within-class scatter matrix as follows:

_{b}

Compared with _{b}

According to _{w}

Then, we employ GSVD [

_{b}, H

_{w}

^{T}

^{T}P

_{A}

Then, face data

In this section, we summarize the complete algorithmic procedures of the proposed approach. In practice, if the dimension of two biometric data

In this section, we provide the nonlinear extensions of two SDA based multimodal feature extraction approaches, which are named KPCA-SDA and KSDA-GSVD. In KPCA-SDA, we first apply Kernel PCA on each single modal before performing SDA. While in KSDA-GSVD, we directly perform Kernel SDA to fuse multimodal data by applying GSVD to avoid the singular problem.

In this subsection, the SDA-PCA approach is performed in a high dimension space by using the kernel trick. We realized the KPCA-SDA in the following steps:

Nonlinear mapping.

Let ∅: ^{d}

Perform KPCA for each single modal database.

For the ^{th}^{th}

According to the kernel reproducing theory [

Substituting _{j}m^{th}^{th}

The solution of

The optimal solutions _{j}_{j}_{1}, _{j}_{2},…, _{j}_{(}_{N-c}_{)})^{T}_{j}

Calculate kernel discriminant vectors in the KPCA transformed space.

By using the KPCA transformed sample set

We can obtain a set of nonlinear discriminant vectors

Construct the nonlinear projection transformation and do classification.

We then construct the nonlinear projection transformation ^{jØ}

After the optimal transform ^{jØ}

In this subsection, the SDA-GSVD is performed in a high dimension space by using the kernel trick. Given two sets of mapped samples
_{b}_{w}

Then, we apply GSVD to calculate the optimal transformation so that the singular problem is avoided. The procedures are precisely introduced in Algorithm 1. When the optimal

Finally, the nearest neighbor classifier with cosine distance is employed to perform classification.

In this section, we compare the proposed multimodal feature extraction approaches with single modal method and several representative multimodal biometric fusion methods. The identification and verification performance of our approaches and other compared methods is evaluated on two face databases and one palmprint database.

Two public face databases (AR and FRGC) and one public palmprint database (PolyU palmprint database) are employed to testify our proposed approaches. The AR face database [

The FRGC database [

The palmprint database [

In order to testify the proposed fusion techniques, in the experiment which we fuse AR database and PolyU palmprint database, we choose 119 subjects from both face and palmprint database, and each class contains 20 samples. Similarly, in the experiment which we fuse FRGC database and PolyU palmprint database, we choose 189 subjects from both face and palmprint database, and each class contains 20 samples. We assume that samples of one subject in the palmprint database correspond to the samples of one subject in the face database. For the AR face database and PolyU palmprint database, we randomly select eight samples from each person (four face samples from AR database and four palmprint samples from PloyU database) for training, while use the rest for testing. For the FRGC face database and PolyU palmprint database, we randomly select six samples from each person (three face samples from FRGC database and three palmprint samples from PloyU database) for training, while use the rest for testing. We run all compared methods 20 times. In our experiments, we consider the Gaussian kernel
_{i}

Firstly, the identification experiments are conducted. Identification is a one-to-many comparison which aims to answer the question of “who is this person?” We compare the identification performance of two proposed approaches,

Verification is a one-to-one comparison which aims to answer the question of “whether the person is one he/she claims to be”. In the verification experiments, we show the receiver operating characteristic (ROC) curves, which plot the false rejection rate (FRR)

In this paper, we present novel multimodal biometric feature extraction approaches using subclass discriminant analysis (SDA). Considering the nonsingularity requirements, we present two ways to overcome this problem. The first is to initially do principle component analysis before SDA, and the second is to employ generalized singular value decomposition (GSVD) to directly obtain the solution. Further, we present the kernel extensions (KPCA-SDA and KSDA-GSVD) for multimodal biometric feature extraction. We perform the experiments on two public face databases (

The work described in this paper was fully supported by the NSFC under Project No. 61073113, the New Century Excellent Talents of Education Ministry under Project No. NCET-09-0162, the Doctoral Foundation of Education Ministry under Project No. 20093223110001, the Qing-Lan Engineering Academic Leader of Jiangsu Province and 333 Engineering of Jiangsu Province.

Illustration of mix-Gaussian distribution of face data and the corresponding palmprint data. In this example, data of two persons are presented. Each contains 12 data, including six faces and s palmprints. We perform PCA on original data for demonstration, and the order of data magnitude is 1e4.

The complete procedures of SDA based multimodal feature extraction.

Demo images of one subject from the AR face database.

Demo images of one subject from the FRGC face database.

Demo images of one subject from the PolyU palmprint database.

Recognition rates of compared methods on AR face and PolyU palmprint databases: (

Recognition rates of compared methods on FRGC face and PolyU palmprint databases: (

ROC curves of all compared methods on AR face and PolyU palmprint databases: (

ROC curves of all compared methods on FRGC face and PolyU palmprint databases: (

Single modal recognition | AR LDA | 75.09 ± 7.39 |

Palmprint LDA | 82.26 ± 3.50 | |

Multimodal recognition | Pixel level fusion [ |
95.35 ± 4.50 |

Parallel feature fusion [ |
92.48 ± 2.61 | |

Serial feature fusion [ |
90.71 ± 3.06 | |

Score level fusion [ |
92.99 ± 2.63 | |

SDA based feature extraction | 96.52 ± 1.16 | |

SDA-GSVD based feature extraction | ||

| ||

( | ||

| ||

| ||

Single modal recognition | AR KDA | 79.50 ± 6.83 |

Palmprint KDA | 83.45 ± 4.47 | |

Multimodal recognition | KPCA-SDA | 98.74 ± 0.45 |

KSDA-GSVD | ||

| ||

( |

Average recognition rates of compared methods on FRGC face and PolyU palmprint databases.

Single modal recognition | FRGC LDA | 78.26 ± 4.53 |

Palmprint LDA | 80.22 ± 3.26 | |

Multimodal recognition | Pixel level fusion [ |
97.21 ± 2.89 |

Parallel feature fusion [ |
94.92 ± 2.17 | |

Serial feature fusion [ |
94.54 ± 1.57 | |

Score level fusion [ |
95.59 ± 4.70 | |

SDA based feature extraction | 98.06 ± 1.09 | |

SDA-GSVD based feature extraction | ||

| ||

( | ||

| ||

| ||

Single modal recognition | AR KDA | 80.44 ± 2.57 |

Palmprint KDA | 81.23 ± 3.26 | |

Multimodal recognition | KPCA-SDA | 98.82 ± 0.32 |

KSDA-GSVD | ||

| ||

( |

The equal error rate (EER) of all compared methods on different databases.

Single modal recognition | Face LDA | 15.45 | 8.13 |

Palmprint LDA | 4.32 | 3.14 | |

Face KDA | 6.13 | 5.72 | |

Palmprint KDA | 8.36 | 10.85 | |

Multimodal recognition | Pixel level fusion [ |
3.95 | 3.25 |

Parallel feature fusion [ |
3.71 | 3.27 | |

Serial feature fusion [ |
7.84 | 4.41 | |

Score level fusion [ |
5.12 | 2.62 | |

SDA based feature extraction | 0.83 | 1.05 | |

SDA-GSVD based feature extraction | 0.72 | ||

KSDA based feature extraction | 0.87 | 1.90 | |

KSDA-GSVD based feature extraction | 0.84 |