Double Quantification of Template and Network for Palmprint Recognition

Abstract

The outputs of deep hash network (DHN) are binary codes, so DHN has high retrieval efficiency in matching phase and can be used for high-speed palmprint recognition, which is a promising biometric modality. In this paper, the templates and network parameters are both quantized for fast and light-weight palmprint recognition. The parameters of DHN are binarized to compress the network weight and accelerate the speed. To avoid accuracy degradation caused by quantization, mutual information is leveraged to optimize the ambiguity in Hamming space to obtain a tri-valued hash code as a palmprint template. Kleene Logic’s tri-valued Hamming distance measures the dissimilarity between palmprint templates. The ablation experiments are tested on the binarization of the network parameter, and the normalization and trivialization of the deep hash output value. The sufficient experiments conducted on several contact and contactless palmprint datasets confirm the multiple advantages of our method.

1. Introduction

Recently, biometric recognition has become a promising and widely used personal identification technology. As a biometric modality, palmprint has many advantages, including rich features, high user acceptance and easy acquisition. Many palmprint recognition methods were proposed and achieved high recognition accuracy. However, with the large-scale applications of biometrics, the sizes of palmprint databases are rapidly increasing. It is urgent to save the storage cost of recognition models and feature templates, and also to accelerate matching speed. Thus, it is necessary to develop fast and light-weight techniques for large-scale palmprint recognition and retrieval [1].

Palmprint recognition based on deep hash network (DHN) has been widely used in many scenarios due to its low storage, low matching computational complexity and high recognition accuracy. However, DHNs ignore model squeezing, i.e., reducing model weight.

The binary deep hash network (B-DHN) has many advantages. The traditional DHN uses format FP32 to represent the weight value, while the weights of B-DHN are binary. Therefore, the parameter storage size of B-DHN is only about 1/32 of that of the traditional DHN. In addition, B-DHN has only “1” and “−1” parameters (weights), which means that the convolution operations can be replaced with bitwise operations. To be specific, B-DHN can replace the traditional multiplication operations with “Logic-And-Gate” and “Logic-XOR-Gate” operations. In other words, in the propagation process of B-DHN, only addition and bit operations are needed instead of multiplication, which is much faster than traditional DHN [2].

In this paper, the templates and network parameters are both quantized for fast and light-weight palmprint recognition. We propose a B-DHN with tri-valued hash codes. Specifically, the weights of the DHN are binarized to accelerate the operations of the network and reduce the model weight. The network compression commonly results in accuracy degradation. Therefore, the outputs of DHN are transformed to tri-valued codes [3] to reduce the ambiguity of the DHN output in Hamming space, which remarkably improves the recognition accuracy.

The contributions of this paper can be summarized as follows:

• The proposed B-DHN has two advantages. On the one hand, the speed of B-DHN is much faster than that of traditional DHN. On the other hand, B-DHN squeezes the network by binarizing the network to reduce the model storage. Thus, B-DHN has the advantages in terms of speed and storage;
• To improve the recognition accuracy of B-DHN, the weights are balanced and standardized by maximizing the information entropy and minimizing the quantization error before the network binarization, which reduces the information loss due to the parameter binarization in forward propagation. The gradient of B-DHN cannot be calculated; therefore, a function, which approximates the gradient to minimize the information loss, ensures sufficient update at the beginning of training and accurate gradient at the end of training;
• In order to reduce the accuracy degradation caused by the squeezing of the DHN, the outputs of DHN are quantized to tri-valued hash codes as the palmprint templates. Mutual information is used to dilute the ambiguity of the output binarization in Hamming space. Kleene Logic’s tri-valued Hamming distance measures the dissimilarity between palmprint templates; thus, the ambiguous intervals have a small weight to improve the recognition accuracy.

The rest of this paper is organized as follows: Section 2 introduces some related works. Section 3 describes the proposed methodology, which includes the B-DHN and the tri-valued hash codes. Section 4 shows the experimental results and discussions. The conclusions and future works are given in Section 5.

The overall structure is shown in Figure 1. BI-CBP denotes binary convolution operation + BN + PReLU, and MI denotes the mutual information transforming of the network outputs to tri-valued hash codes.

Figure 1. Overall structure.

2. Related Works

2.1. Palmprint Recognition

Palmprint recognition typically includes four steps, namely, image acquisition, image preprocessing, feature extraction and matching. Palmprint recognition methods are reviewed as follows.

2.1.1. Local Texture Coding Methods

Filters are commonly used to extract texture features, such as the magnitude and phase of the filter response, and then the features are encoded by some specific rules. The single-direction approaches preserve only single-direction texture. Zhang et al. [4] proposed Palmprint Code (PalmCode), which used 45° Gabor filter to process the image and binarized the response value. Kong et al. [5] filtered the images by six Gabor filters along different directions (0°, 30°, 60°, 90°, 120°, 150°), and coded the winner (dominant) direction index of the filter corresponding to the minimum response of each pixel. Wei et al. [6] used the modified finite Radon transform filter to extract palmprint features and a “winner-take-all rule” to code each pixel. In matching phase, a pixel-to-region template matching method was used to further improve accuracy.

In order to fully utilize the palmprint texture, multi-directional approaches were proposed. Since fusion can improve the accuracy effectively [7,8], Fei et al. [9] used multi-directional Gabor filters for feature extraction, and encoded the direction index corresponding to the top-2 response values. Guo et al. [10] encoded the responses of the multi-directional Gabor filters and fused the matching scores at score level. If the absolute value of the response is closer to 0, the response is regarded as worse. The method removed the 8% worst response entries with a mask matrix and ignored them in the matching phase. Xu et al. [11] encoded the most relevant response and the weighted responses of two adjacent directions. They also used a Gaussian filter to process the image, which reduced the noise effect and further improved the performance.

Some studies on the downsampling were proposed for coding methods. Yang et al. [12] allowed all pixels in each block to have equal voting rights to extract stable features. Leng et al. [13] selected the best impact pixel from the 16 candidates in each block. Yang et al. [14] conducted the downsampling in uniformly-spaced windows, which avoided the small distance and strong correlation between the adjacent preserved pixels.

2.1.2. Deep Learning-Based Methods

Deep learning-based palmprint recognition methods typically use a specific method, such as convolutional neural network (CNN), for feature extraction and classification. Svoboda et al. [15] trained AlexNet to expand the separability between the genuine and impostor distributions, but the training required supervision. Wen et al. [16] proposed a new loss function to increase the distance between classes. Inspired by this, Zhong and Zhu [17] proposed a concentrated large marginal cosine loss, which improved intra-class tightness. Matkowski et al. [18] proposed a CNN framework for palmprint recognition in an uncontrolled environment; they used two sub-networks for segmenting the region of interest (ROI) and extracting the features, respectively. Chai et al. [19] performed gender soft biometric network for pre-training, and then trained the network for palmprint classification. Xu et al. [20] used soft biometrics in multi-task pre-training for palmprint recognition. Du et al. [21] proposed a CNN-based regularized adversarial domain cross-domain recognition model. Liu et al. [22] developed a soft-shift triplet loss function for learning to distinguish palmprint features using a fully convolutional network.

2.2. Deep Hash Network

DHNs combine the advantages of hashing algorithms and CNNs. On the one hand, DHNs have the same robust feature extraction ability as CNN. On the other hand, DHNs combine CNNs with hash encoding, so DHNs reduce the storage space and speed up the matching/retrieval speed. The procedure of DHNs can be summarized as: a CNN is trained for extracting low-dimensional feature, and then the features are transformed into binary codes.

Chen et al. [23] proposed a discriminative spectrum hash to compact palmprint representation. Cheng et al. [24] combined supervised hashing and deep convolutional features for palmprint recognition. Zhong et al. [25] extracted palmprint and dorsal hand vein features using DHN and feature map matching for hand-based multi-biometric recognition. Zhong et al. [26] used a DHN for palmvein recognition with an equal error rate (EER) close to 0% on the near infrared (NIR) spectral dataset. Li et al. [27] used softmax classification loss and improved ternary loss to learn the hash code and maintain consistency with high-dimensional features. Liu et al. [28] used a deep self-supervised hash to generate pseudo labels, and used a DHN to generate hash codes. Wu et al. [29] used a DHN to generate hash codes and feature selection to obtain more compact deep hash codes.

This paper proposes double quantification of template and network for palmprint recognition. A DHN is compressed to a B-DHN to speed up the network speed and reduce the network weight. At the same time, the outputs of network have low storage cost and fast matching/retrieval speed.

3. Methodology

This section describes the proposed B-DHN for tri-valued codes. Firstly, the B-DHN is specified, which includes balanced normalized quantization for parameter binarization and approximation function for gradient calculation. Next, the quantization for the tri-valued hash codes is described, where the ambiguous interval of the deep hash value is labeled by a state. The DHN configuration in this paper is shown in Table 1.

Table 1. Network parameter configuration.

Layer	Configuration
Conv1	Filter 16 × 3 × 3, st.4, pad 0, BN, PReLU
Max_pool	Filter 2 × 2, st.1, pad 0
Conv2	Filter 32 × 5 × 5, st.2, pad 2, BN, PReLU
Max_pool	Filter 2 × 2, st.1, pad 0
Conv3	Filter 64 × 3 × 3, st.1, pad 1, PReLU
Conv4	Filter 64 × 3 × 3, st.1, pad 1, PReLU
Conv5	Filter 128 × 3 × 3, st.1, pad 1, PReLU
Max_pool	Filter 2 × 2, st.1, pad 0
Full6	Length 2048
Full7	Length 2048
Full8	Length 128

3.1. Binary Deep Hash Networks

3.1.1. Binary Convolution and Approximation Function

• Binary convolution

To reduce the information loss in a B-DHN, the network parameters are balanced and normalized before binarization as:

$$Wstd=\frac{W^{\prime }}{\sigma \left(W^{\prime }\right)}$$

(1)

where $\sigma (.)$ is the standard deviation, and $W^{\prime }$ is:

$$W^{\prime }=W-\stackrel{-}{W}$$

(2)

where $W$ is the initial weight value, and $\stackrel{-}{W}$ is the mean value of $W$.

Final, binary convolution can be formulated as:

$$Z_b=Bco\left(sign\right(A),sign(Wstd\left)\right)$$

(3)

where $A$ denotes the input activation vector computed by the previous network layer, and $Bco$ (.) denotes binary convolution operation, which replaces the traditional multiplication operation with “Logic-XOR-Gate” operation. Contrary to traditional convolution operations, the binary convolution conducts bitwise operations.

2.

Approximation function

Since the gradient of binary network cannot be calculated directly, approximation function is used to replace binarization function. Approximation function of the parameter $x$ is:

$$g\left(x\right)=ktanh\left(tx\right)$$

(4)

$$t=T_1{10}^{\frac{i}{N}\times log\frac{T_2}{T_1}}$$

(5)

$$k=max(\frac{1}{t},1)$$

(6)

where $i$ is the current epoch index, and $N$ is the maximum epoch number, $T_1={10}^{-1}$ and $T_2={10}^1$.

3.1.2. Loss Function of B-DHN

The loss function is divided into two parts. The first part is distance loss, which keeps the feature vectors of the same class close together and the feature vectors of the different classes far away. The second part is quantization loss; a regular term is added to the real-valued network output in the last layer.

• Distance loss function

In order to decrease the intra-class distance and enlarge the inter-class distance, the intra-class distance is formulated as:

$$S\left(f_i,f_j\right)={\frac{1}{2}l}_{ij}HD\left(f_i,f_j\right)$$

(7)

where $f_i$ and $f_j$ denote two binary codes (templates) of the input images of the identical class. $HD\left(.\right)$ denotes the Hamming distance. The inter-class distance is:

$$\stackrel{-}{S}\left(f_i,f_j\right)=\frac{1}{2}\left(1-l_{ij}\right)max\left(T-HD\left(f_i,f_j\right),0\right)$$

(8)

If $f_i$ and $f_j$ belong to the identical class, $l_{ij}$ = 1; otherwise $l_{ij}$ = 0. $T$ denotes the distance threshold. In the experiment, $T$ is set to 90 because each output is coded as 128 bits.

By combining the above two formulae, the distance loss function is:

$$R_s\left(f_i,f_{j,}{l}_{ij}\right)=\sum _{i=1}^M\sum _{j=1}^M\left[S\left(f_i,f_j\right)+\stackrel{-}{S}\left(f_i,f_j\right)\right]$$

(9)

where $M$ is the number of palmprint images in the training set.

2.

Quantization loss function

To reduce quantization error, each entry in the output should be close to “+1” or “−1”, so the quantization function is:

$$R_Q={\sum }_{i=1}^M\frac{1}{2}\left({‖1-\left|f_i\right|‖}_2\right)$$

(10)

where $\left|f_i\right|$ denotes the absolute value of $f_i$, and ${‖.‖}_2$ denotes the L2−Norm of the vector.

Ultimately the loss function is:

$$Loss=\alpha R_s+R_Q$$

(11)

where $\alpha$ is the combination factor.

3.2. Tri-Valued Hash Codes

It is worth noting that the feature extraction of DHNs uses binarized output, which is fine for classification and can still achieve relatively high accuracy even if binarized outputs ignore the ambiguous interval in Hamming space. In this paper, the tri-valued quantification of hash codes fully considers ambiguous interval, so the accuracy is better than that of DHNs. To reduce quantization error, the network outputs are balanced before quantization.

$$Xstd=\frac{X^{\prime }}{\sigma \left(X^{\prime }\right)}$$

(12)

$$X^{\prime }=X-\stackrel{-}{X}$$

(13)

where $\stackrel{-}{X}$ is the mean of network output.

Two thresholds need to be determined for coding network output by finding the largest mutual information. Mutual information is:

$$I(A,B)=\iint P(A,B)log\frac{P(A,B)}{P\left(A\right)P\left(B\right)}$$

(14)

where $A$ and $B$ are the distributions of two sets of network outputs, respectively, and the threshold selection is formulated as:

$$E_{t1,t2}=I(A^{\prime },B^{\prime })$$

(15)

$$A^{\prime }=\left\{x\right|A\left(x\right)\},x\notin (\beta ,\delta ) and B^{\prime }=\{x\left|B\right(x\left)\right\},x\notin (\beta ,\delta ),\beta ,\delta ϵ(-\mathrm{4,4})$$

(16)

$$t1,t2=argmax\left(E_{t1,t2}\right)$$

(17)

where $\beta <\delta$ and both are integers. $t1$ and $t2$ denote two thresholds. After obtaining the two thresholds, x, a bit of the output, is transformed to a tri−valued digit as:

$$Tcode(x)=\left\{\begin{array}{c}-1,x\le t1\\ 0,t1\le x\le t2\\ 1,x\ge t2\end{array}\right.$$

(18)

The tri-valued Hamming distance table of Kleene logic [30,31] is shown in Table 2, where “0” labels the ambiguous state. The Kleene logic table is simplified because only two calculators, $\neg$ and $\leftrightarrow$, are used for distance calculation. For two tri-valued digits, $a$ and $b\in \{-\mathrm{1,0},1\}$, the tri-valued Hamming distance is:

$$THD(a,b)=\gamma (\neg (a\leftrightarrow b+1\left)\right)$$

(19)

where $\gamma$ is the weight factor, which is generally set to 0.5, i.e., the weight is 0.5 for ambiguous state.

Table 2. Kleene Logic Table.

		A ↔ B			¬A
B		−1	0	1
A	−1	1	0	−1	0
	0	0	0	0	0
	1	−1	0	1	−1

4. Experimental Results and Discussions

This section introduces the palmprint databases and describes the experiments. It also presents the ablation experiments on the balanced and normalized network parameters, the normalized network output, and the tri-valued quantization.

4.1. Dataset and Experimental Environment Setup

In this paper, four palmprint databases, including one contact database and three non-contact databases, are used for performance evaluation and comparison. The original images are pre-processed to ROI with the size of 128 × 128 [32]. The details of the databases are described in Table 3. The multi-spectral database contains four spectrum sub-datasets, namely, Blue, Red, Green and NIR.

Table 3. Details of the palmprint database.

Databases	PolyU	IITD	Multispectral	Tongji-Print
Collection	Touch	Touchless	Touchless	Touchless
Number of class	378	460	500	600
Number of samples per class	20	5	12	20
Total number of samples	7560	2300	24,000	12,000

• PolyU database [33]. A total of 7752 images belong to 386 palms, each palm containing around 20 images. The images are all acquired with a contact device. There are, in total, 30,042,876 matchings, including 74,068 genuine matchings and 29,968,808 imposter matchings;
• Multispectral database [34]. The images are acquired with contact devices from different spectral environments. Each spectral database contains 6000 palm images. There are, in total, 1,799,700 matchings, including 33,000 genuine matchings and 1,796,400 imposter matchings;
• IITD database [35]. There are 2601 hand images captured with contactless device from 230 individuals (460 palms). Each palm has around five images. Contactless acquisition usually contains stronger noise;
• Tongji-print database [36]. It consists of 12,000 images of 300 individuals (600 palms) acquired with a contactless device in two sessions. In each session, 10 images of each palm are acquired. There are, in total, 1,799,700 matchings, including 2700 genuine matchings and 17,970,000 imposter matchings.

The image samples of these databases are shown in Figure 2. The four databases are acquired in different ways; thus, the experiments on these databases can better reflect the advantages of the our method. To ensure the stability of the experiments, the categories with incomplete sample number are removed.

Figure 2. Samples in different palmprint databases.

The system configuration is: AMD Ryzen 5 3600 6-Core Processor 3.60 GHz, NVIDIA GPU GTX3060ti, 16 GB RAM and Windows 10 OS. All experiments are conducted on PyTorch.

4.2. Ablation Experiments

• Balanced and normalized network parameters

Balanced and normalized network parameters can maximize the information entropy of binary weights and binary activations by adjusting the distribution of network weights. Since an explicit balance operation is used before the binarization, the binary weights of each layer in the network have maximum information entropy. Binary activations affected by binary weights also have maximum information entropy.

2.

Balanced network output

Before balanced network output, the distribution of network output is concentrated around “0”, i.e., many values are in the ambiguous interval, which is not conducive to the quantization operation. The balanced network outputs have maximum information entropy to reduce the ambiguity of the network output.

3.

Tri-valued quantization

Traditional binarization ignores the ambiguity in Hamming space, and direct binarization with a threshold or a function can lead to misclassification. Specifically, logical “true” feature values are misclassified as logical “false”, and vice versa. This uncertainty is directly reflected in the distance matching process, where the intra-class distance is too large and the inter-class distance is too small [37].

Table 4 shows the ablation experimental results on the PolyU database. The balanced and normalized network parameters greatly improve the accuracy. Balanced network output and tri-valued quantization are both helpful to accuracy improvement.

Table 4. Ablation experiments on PolyU.

Balanced and Normalized Network Parameters	Balanced Network Output	Tri-Valued Quantization	EER (%)
-	-	-	3.0431
√	-	-	0.0913
-	√	√	1.8691
√	-	√	0.0763
√	√	-	0.0850
√	√	√	0.0673

4.3. Comparison Experiments

The comparison experimental results are shown in Table 5. The proposed method is compared with many SOTA methods, including local texture coding methods, such as PalmCode [4], ordinal code (OrdinalCode) [38], fusion code (FusionCode) [39], competitive code (CompCode) [5], robust line orientation code (RLOC) [6], half-orientation code (HOC) [40], double-orientation code (DOC) [9], discriminative competitive code (DCC) [11], discriminative and robust competitive code (DRCC) [11] and binary orientation co-occurrence vector (BOCV) [10]; deep learning-based methods, such as palmprint network (PalmNet) [41]; and deep hash-based methods, such as deep hash codes (DHC) [29] and deep tri-valued code (DTC) [3].

Table 5. EER (%) of various methods on various databases.

	PolyU	IITD	Tongji
PalmCode	0.3500	5.4500	0.1100
OrdinalCode	0.2300	5.5000	0.1600
FusionCode	0.2400	6.2000	0.0731
CompCode	0.1200	5.5000	0.1100
RLOC	0.1300	5.0000	0.0253
HOC	0.1600	6.5500	0.0954
DOC	0.1800	6.2000	0.0431
DCC	0.1500	5.4900	0.0506
DRCC	0.1800	5.4200	0.0308
BOCV	0.0856	4.5600	0.0056
DHPN	0.0456	3.7310	0.0694
PalmNet	0.1110	4.2040	0.0332
DHC	0.0513	3.1180	0.0001
DTC	0.0302	2.9270	0.0000
Ours	0.0673	3.7960	0.0075

The lower the EER is, the stronger the discrimination. Table 6 shows the EERs of different methods on multispectral databases. Figure 3 and Figure 4 show the receiver operating characteristic (ROC) curves. The proposed method can obtain the optimal results that are close to the best results, but our method has more advantages, including low storage cost of template and model, and low computational complexity of template generation and matching.

Table 6. EER (%) of various methods on multispectral database.

	Blue	Green	Red	NIR
PalmCode	0.2800	0.2500	0.2300	0.2000
OrdinalCode	0.1600	0.1500	0.0720	0.1100
FusionCode	0.3100	0.1900	0.1200	0.1700
CompCode	0.0911	0.1100	0.0357	0.0579
RLOC	0.0799	0.0855	0.0443	0.0629
HOC	0.1800	0.1600	0.1000	0.0839
DOC	0.1300	0.1200	0.0584	0.0501
DCC	0.1100	0.0979	0.0450	0.0575
DRCC	0.1100	0.0927	0.0659	0.0563
BOCV	0.0358	0.0593	0.0241	0.0261
DHPN	0.0213	0.0352	0.0369	0.0020
PalmNet	0.0178	0.0087	0.0366	0.0871
DHC	0.0000	0.0000	0.0000	0.0000
DTC	0.0000	0.0000	0.0000	0.0000
Ours	0.0018	0.0003	0.0087	0.0159

Figure 3. ROC curves of various methods on various databases.

Figure 4. ROC curves of various methods on multispectral database.

Table 7 compares the storage cost of template. The storage cost of the proposed method is the lowest, so the proposed method is highly efficient in matching/retrieval phase.

Table 7. Storage cost (bit).

Method	Storage	Method	Storage
PalmCode	2048	HOC	2048
OrdinalCode	3072	DOC	2048
FusionCode	2048	DRCC	2048
CompCode	3072	BOCV	6144
RLOC	6144	Ours	128

For the deep learning-based methods, the proposed method still has high accuracy. For deep hash-based methods, the accuracy of the proposed method is slightly worse than those of DHC and DTC. The same phenomenon can be found in Figure 3 and Figure 4. However, the proposed method is compressed, its computational complexity and model storage are much smaller, as shown in Table 8. Parameter volume (Params) is used to measure model storage, and computational complexity is measured by multiply-ACCumulate operation (MACC) and floating-point operations (FLOPs). The weight and activation (W/A) are also shown. The model storage of the proposed method is only 6.67 MB, about 1/32 of those of DTC and DHC. At the same time, the proposed method conducts convolution operation in bitwise mode instead of float mode, which is far more computationally efficient than DTC and DHC.

Table 8. Computational complexity and model storage.

Method	Bit-Width (W/A)	Operation Type	Params (MB)	FLOPs (M) *	MACC (M)
DHPN (Feature Extraction + PCA)	32/32	Float	527.76	30,816.89	13,621.10
DHN	32/32	Float	213.49	176.58	88.29
DTC (Network + Quantification)	32/32	Float	213.49	176.59	88.29
Ours	1/1	Bitwise	6.67	-	88.29

* Our method does not have floating operations.

On the NIR dataset, the accuracy of the proposed method degrades due to the harsh acquisition environment, and we will further improve the algorithm on this dataset in the future.

5. Conclusions and Future Works

In this paper, a B-DHN for tri-valued hash codes is proposed for the double quantification of template and network in fast and light-weight palmprint recognition. Balanced and normalized network parameters are used to reduce the information loss after weighting and activation binarization. Approximation function can better represent the quantization function, ensuring the network to run smoothly. The model size of our method is only 1/32 of that of a DHN, which remarkably reduces the network storage cost. At the same time, bitwise operations instead of float operations remarkably reduce the network computational complexity. The tri-valued quantization for hash codes improves the accuracy by reducing the ambiguity interval. In future work, we will try to overcome the sensitivity of the network to illumination, especially NIR.