Feature extraction from images or image regions is a key step for image recognition and video analysis problems. Recently, matrix-based feature representations [1–6] have been developed and employed for feature extraction. Tuzel et al. [3] introduced the region covariance matrix (RCM) as a new image region descriptor and have applied it to object detection and texture classification. RCM is a covariance matrix of basic features extracted from a region. The diagonal entries of the covariance matrix represent the variance of each feature, while the nondiagonal entries represent the respective correlations. Using RCM as region descriptor has several advantages. Firstly, RCM provides a natural fusion method because it can fuse multiple basic features without any normalization or weight operations. Secondly, RCM can be invariant to rotations. Thirdly, its computational cost does not depend on the size of the region. Due to these advantages, RCM has been employed to detect and track objects [3,5], and has achieved promising results. The RCM in [3] and [5] were constructed using the basic features including the pixel locations, color values and the norm of the first and second order derivatives. However, directly employing RCM for human face recognition cannot achieve higher recognition rates. In order to improve face recognition rates, Pang et al. [4] proposed the Gabor-based RCM (GRCM) method using pixel locations and Gabor features to construct region covariance. As Gabor features can carry more discriminating information, GRCM displayed better performance. Subsequently, they also proposed a kernel Gabor RCM (KGRCM) method [7] to capture the higher order statistics in the original low-dimensional space. Their experimental results have demonstrated that the KGRCM can improve the classification performance. Recently, KGRCM has been applied to object detection and tracking [6]. The nonlinear descriptor can capture nonlinear relationships within image regions due to the usage of nonlinear region covariance matrix.

However, the previous methods based on RCM consider each pixel in the training image to be contributing equally when reconstructing the RCM, i.e., the contribution of each pixel is usually set to be 1/N², where N is the number of pixels in a local region. However, this assumption of equal contribution does not hold in real-world applications because it is possible that different pixels in different image parts may have different discriminative powers. For example, pixels at important facial features such as eyes, mouth, and nose should be emphasized and others such as cheek and forehead should be deemphasized.

Motivated by the above-mentioned reasons, we hence propose in this paper a weighted region covariance matrix (WRCM) to explicitly exploit the different importance of each pixel of a sample. WRCM can only extract linear face features. However, by using nonlinear features it can achieve higher performance for face recognition tasks [7–9]. To further preserve nonlinear features, we develop the kernel Gabor-based weighted region covariance matrix (KGWRCM). Experimental results on the ORL Face database [10], the Yale Face database [11] and the AR database [12] show that the KGWRCM algorithm outperforms the RCM, the WRCM, the RCM with Gabor features (GRCM) [4], the KRCM with Gabor features (KGRCM) [7], and the conventional KPCA [9], Gabor + PCA [13], and Gabor +LDA [13] algorithms in terms of the recognition rate.

The RCM [3] is a matrix of covariance of features computed inside a region of an image. Let F be a two-dimensional image size of h × w, where w and h are the height and width of the image region. The number of pixels in image region is N = h × w. Define a mapping ϕ that maps a pixel (k, l) of F onto the d dimensional feature vector x_i: (1) ϕ ( F , k , l ) = x i ∈ Φ d

As a result there are N d-dimensional feature vectors (x_i)_{i = 1,…},_N. For the intensity image, the feature mapping function ϕ is defined by pixel locations, gray values and the norm of the first and second order derivatives of the intensities with respect to k and l: (2) ϕ ( F , k , l ) = [ k l F ( k , l ) | F k | | F l | | F k k | | F l l | ]

The image region can then be represented by a d × d covariance matrix of the basic feature vectors x_i: (3) C = 1 N ∑ i = 1 N ( x i - u ) ( x i - u ) Twhere μ is the mean of the feature vectors x_i: (4) u = 1 N ∑ i = 1 N x i

Equation (3) can also be expressed by following equation: (5) C = 1 2 1 N × N ∑ i = 1 N ( x i - x j ) ( x i - x j ) T

The computation process is given in Appendix A.

Based on the feature vectors x_i, the d × d weighted region covariance matrix of the image region is defined as follows: (6) C W = 1 2 ∑ j = 1 N ∑ i = 1 N ( x i - x j ) 2 S i j = ∑ j = 1 N ∑ i = 1 N x i S i j ( x i ) T - ∑ j = 1 N ∑ i = 1 N x i S i j ( x j ) T = ∑ i = 1 N x i D i i ( x i ) T - X S X T = X D X T - X S X T = X ( D - S ) X T = X L X Twhere the matrix S is a similarity matrix [14], which is chosen as: (7) S i j = exp ( - ‖ x i - x j ‖ 2 / σ )with a value ranging from 0 to 1, and σ is a suitable constant. D_ii is a diagonal matrixes whose entries are column or row sums of S, D i i = ∑ j = 1 N S i j. L = D − S is a matrix of N × N.

Comparing Equations (5) and (6), we can see that the WRCM is just the RCM if S_ij = 1/N², which implies that RCM is a special case of the WRCM method. However, as all the weights in RCM are 1/N², RCM cannot exploit the different importance of each pixel of a sample. On the other hand, the WRCM can assign different weights for each pixel of a sample, so it can preserve more discrimination information than RCM.

As C_W in Equation (6) is a matrix-form feature, the commonly used distances are not used. The generalized eigenvalue based distance proposed by Forstner [15] is hence used to measure the distance/dissimilarity between the WRCMs C W p and C W g: (8) D ( C W p , C W g ) = ∑ i = 1 c ln 2 λ i ( C W p , C W g )where λ₁,…,λ_c are the generalized eigenvalues of covariance C W p and C W g computed from: (9) λ i C W p v i = C W g v i i = 1 , … , c

To preserve the local and global patterns, similar to [3,4], we represent a face image with five WRCMs from five different regions (R₁, R₂, R₃, R₄, and R₅) (Figure 1). The five WRCMs (C_W1, C_W2, C_W3, C_W4, and C_W5) are constructed from five different regions. As C_W1 is the weighted region covariance matrix of the entire image region R₁, it is a global representation of the face. The C_W2, C_W3, C_W4, and C_W5 are extracted from four local image regions (R₂, R₃, R₄, and R₅), so they are part-based representations of the face.

After obtaining WRCMs of each region, it is necessary to measure the distance between the gallery and probe sets. Let C W p and C W g be WRCMs from the gallery and probe sets. The distance between a gallery WRCM and a probe one is computed as follows: (10) d ( C W p , C W g ) = min j [ ∑ i = 1 5 D ( C W i p , C W i g ) - D ( C W j p , C W j g ) ] = ∑ i = 1 5 D ( C W i p , C W i g ) - max j [ D ( C W j p , C W j g ) ]

To generalize WRCM to the nonlinear case, we use a nonlinear kernel mapping x ∈ Φ^d → ϕ(x) ∈ Ω to map the feature data x ∈ Φ^d into a higher dimensional subspace. Then a linear WRCM is performed to preserve intrinsic geometric structures in subspace Ω. Suppose that R₁ and R₂ are two rectangular regions in the gallery and probe set images, respectively. Let m and n be number of pixels located in regions R₁ and R₂, respectively. ϕ(x) and ϕ(y) are the higher dimensional features extracted from regions R₁ and R₂, where, ϕ(X) = [ϕ(x₁), ϕ(x₂),…, ϕ(x_m)] and ϕ(Y) = [ϕ(y₁), ϕ(y₂),…, ϕ(y_n)]. Let C ϕ W p and C ϕ W g be the kernel weighted region covariance matrices of regions R₁ and R₂, respectively. C ϕ W p and C ϕ W g are computed as follows: (11) C ϕ W p = ∑ j = 1 m ∑ i = 1 m ϕ ( x i ) S i j ∗ ϕ T ( x i ) - ∑ j = 1 m ∑ i = 1 m ϕ ( x i ) S i j ∗ ϕ T ( x j ) = ∑ i = 1 m ϕ ( x i ) D i i ∗ ϕ T ( x i ) - ϕ ( X ) S ∗ ϕ T ( X ) = ϕ ( X ) D ∗ ϕ T ( X ) - ϕ ( X ) S ∗ ϕ T ( X ) = ϕ ( X ) ( D ∗ - S ∗ ) ϕ T ( X ) = ϕ ( X ) L ∗ ϕ T ( X )where S i j ∗ = exp ( - ‖ x i - x j ‖ 2 / t ), D i i ∗ = ∑ j = 1 n S i j ∗, and L* = D* − S*: (12) C ϕ W g = ∑ j = 1 n ∑ i = 1 n ϕ ( y i ) S i j # ϕ T ( y i ) - ∑ j = 1 n ∑ i = 1 n ϕ ( y i ) S i j # ϕ T ( y j ) = ∑ i = 1 n ϕ ( y i ) D i i # ϕ T ( y i ) - ϕ ( Y ) S # ϕ T ( Y ) = ϕ ( Y ) D # ϕ T ( Y ) - ϕ ( Y ) S # ϕ T ( Y ) = ϕ ( Y ) ( D # - S # ) ϕ T ( Y ) = ϕ ( Y ) L # ϕ T ( Y )where S i j # = exp ( - ‖ y i - y j ‖ 2 / t ), D i i # = ∑ j = 1 n S i j #, and L^# = D^# − S^#.

Hence Equation (9) can be written as follows: (13) ϕ ( X ) L ∗ ϕ T ( X ) V = λ ϕ ( Y ) L # ϕ T ( Y ) V

As any eigenvector can be expressed by a linear combination of the elements, there exist coefficients α_i (i = 1,2,…,m) and β_j (j = 1,2,…,n) such that: (14) V = ∑ i = 1 m α i ϕ ( x i ) + ∑ j = 1 n β j ϕ ( y j ) = ϕ ( X ) α + ϕ ( Y ) βwhere α = [α₁,α₂,…α_m]^T and β = [β₁,β₂,…β_n]^T.

Combining Equations (13) and (14), the generalized eigenvalue task in Equation (13) can be expressed in the form of block matrices: (15) [ K ( X , X ) L ∗ K ( X , X ) K ( X , X ) L ∗ K ( X , Y ) K ( Y , X ) L ∗ K ( X , X ) K ( Y , X ) L ∗ K ( X , Y ) ] [ α β ] = λ [ K ( X , Y ) L # K ( Y , X ) K ( X , Y ) L # K ( Y , Y ) K ( Y , Y ) L # K ( Y , X ) K ( Y , Y ) L # K ( Y , Y ) ] [ α β ]

The detailed derivation of Equation (15) is given in Appendix B.

We defined matrices U, A, and B as (16) U = [ α β ] (17) A = [ K ( X , X ) L ∗ K ( X , X ) K ( X , X ) L ∗ K ( X , Y ) K ( Y , X ) L ∗ K ( X , X ) K ( Y , X ) L ∗ K ( X , Y ) ] (18) B = [ K ( Y , Y ) L # K ( Y , Y ) K ( Y , Y ) L # K ( Y , X ) K ( X , Y ) L # K ( Y , Y ) K ( X , Y ) L # K ( Y , X ) ]

Equation (15) can be rewritten as: (19) A U = λ B U

When A is positive definite, the generalized eigenvalues are obtained through solving the following eigenvalue problem: (20) λ B A - 1 U = U

However, in many cases, A is a singular matrix, we hence incorporate a regularization parameter u > 0 on both sides, respectively: (21) ( A + u I ) U = λ ( B + u I ) Uwhere I is an identity matrix. When u is large enough, (B + uI) is positive definite. Equation (21) becomes a standard eigenvalue problem: (22) ( A + u I ) ( B + u I ) - 1 U = λ U

Based on eigenvalues obtained by Equation (9) or Equation (22), we compute the distance between the two image regions R₁ and R₂ using Equation (8).

In Equation (2), these features such as pixel locations (k,l), intensity values and the norm of the first and second order derivatives of the intensities with respect to k and l are effective for tracking and detecting objects. However, their discriminating ability is not strong enough for face recognition [4]. To further improve the performance, Gabor features are added to the feature space. A 2-D Gabor wavelet kernel is the product of an elliptical Gaussian envelope and a complex plane wave, defined as: (23) ψ u , v ( z) = ‖ k u , v ‖ 2 σ 2 e - ‖ k u , v ‖ 2 ‖ z ‖ 2 / 2 σ 2 ( e i k u , v z - e - σ 2 / 2 )where u and v define the orientation and scale of the Gabor kernels, z = (x, y), ║•║ denotes the norm operator, and the wave vector k_u,v is defined as follows: (24) k u , v = k v e i ϕ uwhere k_v = k_max/f^v and ϕ_u = πu/8. k_max is the maximum frequency, and f is the spacing factor between kernels in the frequency domain. We use Gabor functions with eight orientations u = {0,…,7} and five scales v = {0,…,4}, making a total of 40 Gabor functions. The value of other parameters follows the setting in [16]: σ = 2π, k max = π 2, f = 2. The Gabor-based features are obtained by the convolution of image F with the 40 Gabor wavelets, using the following formula: (25) g uv ( z ) = | F ( z ) ∗ ψ u , v ( z ) |where, * denotes the convolution operator, and |•| is a magnitude operator.

Therefore, a feature mapping function based on Gabor features is obtain by: (26) ϕ ( F , k , l ) = [ k l F ( k , l ) g 00 ( k , l ) , g 01 ( k , l ) , … , g 74 ( k , l ) ]

As the Gabor wavelet representation can capture salient visual properties such as spatial localization, orientation selectivity, and spatial frequency characteristic, Gabor-based features can carry more important information. The proposed KGWRCM method can be briefly summarized as follows:

partition a face image into five regions (R₂, R₃, R₄, and R₅), and extract basic features of five regions using Equation (26).

We tested the GKWRCM algorithm on the ORL Face database [10], the Yale Face database [11] and AR Face database [12]. The ORL Face database comprises of 400 different images of 40 distinct subjects. Each subject provides 10 images that include variations in pose and scale. To reduce computational cost, each original image is resized to 56 × 46 by the nearest-neighbor interpolation function. A random subset with five images per individual is taken with labels to comprise the training set, and the remaining constructs the testing set. There are totally 252 different ways of selecting five images for training and five for testing. We select 20 random subsets with five images for training and five for testing.

The Yale face database contains 165 grayscale images with 11 images for each of 15 individuals. These images are subject to expression and lighting variations. In this recognition experiment, all face images with size of 80 × 80 were resized to 40 × 40. Five images of each subject were randomly chosen for training and the remaining six images were used for testing. There are hence 462 different selection ways. We select 20 random subsets with five images for training and six for testing.

The AR database consists of over 4,000 images corresponding to 126 people's faces (70 men and 56 women). These images include more facial variations, including illumination change, and facial occlusions (sun glasses and scarf). For each individual, 26 pictures were taken in two separate sessions and each section contains 13 images. In the experiment, we chose a subset of the data set consisting of 50 male subjects and 50 female subjects with seven images for each subject. The size of images are 165 × 120. We select two images for training and five for testing from the seven images. There are 21 different selection ways. Figure 2 shows some examples of the first object in each database used here.

In our experiment, all images are normalized to zero mean and unit variance. We compare the developed WRCM and KGWRCM with the RCM, the RCM with Gabor features GRCM [4], the KGRCM [7] and the conventional methods including KPCA [11], Gabor + PCA [12], and Gabor +LDA [12]. For GPCA and the PCA stage of KPCA, we keep 99% information to determine the number of the principal components. For the PCA stage of Gabor+LDA, we selected the number of principal components as M − c, where M is the number of training samples and c is the number of classes (M = 200 and c = 40 for ORL database, and M = 55 and c = 11 for Yale database, and M = 200 and c = 100 for Yale database). For KPCA, KGRCM, and the proposed KGWRCM, a Gaussian kernel function is used as kernel. We select their parameters for following algorithms with cross-validation method. Their parameters are summarized as follows: (1) parameter σ for the WRCM and KGWRCM methods. (2) the kernel parameters for the KPCA, KGRCM, and KGWRCM methods. In all these experiments, the classifiers of nearest neighborhood are employed. The performance of all methods is evaluated by the mean and standard deviation (std) of the recognition accuracies on 20 data sets for ORL and Yale databases, and the 21 data sets for AR database.

The average recognition accuracies on the ORL, Yale and AR databases are shown in Tables 1, 2 and 3, respectively. For the ORL face database and the Yale face database, the proposed KGWRCM method achieves 99.21% and 79.20% mean recognition accuracy which is higher than that of other methods. For the AR database, the proposed KGWRCM method achieves 95.95% mean recognition accuracy, which is 4.15% higher than that of KGRCM and much higher than that of other methods.

These results clearly show that the proposed KGWRCM method can capture more discriminative information than other methods for face recognition. Particularly KGWRCM and WRCM outperform KGRCM and RCM, which implies that the weighted approaches can better emphasize more important parts in faces and deemphasize the less important parts, and also preserve discriminated information for face recognition.

In this paper, an efficient image representation method for face recognition called KGWRCM is proposed. Considering that some pixels in face image are more effectual in representing and recognizing faces, we have constructed KGWRCM based on weighted score of each pixel within a sample to duly emphasize the face features. As the weighted matrix can carry more important information, the proposed method has shown good performance. Experimental results also show that the proposed KGWRCM method outperforms other approaches in terms of recognition accuracy. However, similar to KGRCM, the computational cost of KGWRCM is high due to the computation of the high dimensional matrix. In future work, an effective KGWRCM method with low computational complexity will be developed for face recognition.