Sweet—An Open Source Modular Platform for Contactless Hand Vascular Biometric Experiments

Abstract

Current finger-vein or palm-vein recognition systems usually require direct contact of the subject with the apparatus. This can be problematic in environments where hygiene is of primary importance. In this work we present a contactless vascular biometrics sensor platform named sweet which can be used for hand vascular biometrics studies (wrist, palm, and finger-vein) and surface features such as palmprint. It supports several acquisition modalities such as multi-spectral Near-Infrared (NIR), RGB-color, Stereo Vision (SV) and Photometric Stereo (PS). Using this platform we collected a dataset consisting of the fingers, palm and wrist vascular data of 120 subjects. We present biometric experimental results, focusing on Finger-Vein Recognition (FVR). Finally, we discuss fusion of multiple modalities. The acquisition software, parts of the hardware design, the new FV dataset, as well as source-code for our experiments are publicly available for research purposes.

1. Introduction

Vascular Biometrics [1,2], or Vein Recognition (VR), offers several advantages [3] such as convenience, high recognition accuracy, and robustness to spoofing over extrinsic biometric modalities such as face, fingerprint, or iris. Most existing FVR devices require the subject’s finger to be in contact with the device. They rely on transmissive Near-Infrared (NIR) illumination [4,5], where the finger is placed between the illuminator and the camera. The NIR light is scattered in the finger-tissue and absorbed by oxygenated hemoglobin in the blood vessels. This design has the advantage that the sensor captures only light that has traveled through the finger and is robust to interference due to external light [6].

Direct contact with a biometric device can be a concern, especially from the point of view of hygiene, which can be critical in environments such as hospitals where transmission of pathogens via surfaces should be mitigated  [7]. Some systems require the finger to be placed in an enclosure which users may find uncomfortable [8].

Reflection-based vascular biometric systems, on the other hand, can be made contactless [9] as by their geometry the illuminator is facing the same direction as the sensor [10]. This family of sensors takes advantage of the penetration depth, typically a few millimeters, of NIR light into the skin tissue which enables capture of the shallow vein network. Unfortunately, this technique yields a signal of a much lower quality and is very sensitive to environmental conditions.

In this work, we present a modular platform named sweet, as shown in Figure 1, aimed at exploring several technologies and sensors with the goal of improving state-of-the-art contactless reflective vascular hand biometrics. Our device is able to acquire images of fingers, palm and wrist in several near-infrared (NIR) wavelengths, for vascular features, as well as in visible light (RGB), for surface features such as fingerprint and palmprint. Moreover, this platform also captures depth information using a pair of NIR cameras with laser pattern projectors and by varying the angle of incidence of the illumination source. This data can be combined to reconstruct the precise shape, texture and reflectivity map, throughout various wavelengths, of the target using Stereo Vision (SV), Photometric Stereo (PS) and 3D image-alignment, yielding high quality original data unavailable, to our knowledge, in commercial hand biometric sensors.

While the primary focus on this article is to introduce the hardware prototype, we also briefly present FVR experiments to study the efficacy of our sweet platform (Some FVR experiments were previously published in ICPR 2024 [11]; they are discussed here to ensure completeness of the presented work). The FVR experiments discussed here are based on CandyFV- new dataset collected from 120 subjects using the sweet platform [12]. While most currently available FVR sensors, including some of the most recent ones, require the user to present one finger at a time, our platform, by design, can capture vein-images of multiple fingers simultaneously by using large Field of View (FoV) optics and high resolution sensors. This, in turn, enables us to combine information from several fingers, to increase the FVR accuracy and reduce the FV recognition HTER to 0.057% [11]. These results demonstrate that the sweet platform offers state-of-the-art FVR performance.

This article is organized as follows: After a discussion about previous research related to the presented work in Section 2, we describe the hardware design for the sweet sensor platform in Section 3. In Section 4 we present the underlying software stack, pre-processing, calibration, SV depth inference and PS. The FVR performance evaluated using a CandyFV dataset collected using the sweet platform are discussed in Section 5. A brief summary and conclusions in Section 6 close this paper.

2. Related Work

Contactless vascular biometric systems [9] have been investigated for more than a decade. Reflective VR systems have been developed for finger [10], palm [13,14], palm and finger  [15], wrist [16] and forehead  [17]. In [18], Yuan and Tang have also proposed combining surface and vascular features such as palm-vein with palm-print.

Multi-spectral biometric systems [19] have been used for a variety of modalities and applications. Spinoulas et al. used color RGB, NIR and Short-Wave Infrared (SWIR) illumination to improve face presentation attack detection (PAD) [20]. Multi-spectral data has been shown to improve iris recognition [21], and fingerprint recognition  [22]. Hao et al. have developed a multi-spectral imaging device for contactless palmprint recognition [23].

Using 3D depth information [24] for face PAD is now common in consumers products such as smartphones. Although Stereo Vision (SV) -based VR methods have been recently proposed [25], these systems are still quite rare. In [26], Kauba et al. proposed a transmissive acquisition technique where the camera and illumination module rotates around the finger, whereas in [27] three cameras surround the finger. In  [28] photometric stereo (PS) is proposed as biometric modality using 3D knuckle patterns on the fingers.

While most apparatus in biometric laboratories are expensive research prototype, the authors in [29] used a Raspberry Pi (RPi) platform for the acquisition computer and in [30] a RPi NoiR camera is employed as a sensor.

Newly proposed FVR algorithms are compared to the state-of-the-art using publicly available datasets such as SDUMLA-HMT [31], MMCBNU_6000  [32], VERA-finger [4], UTFVP [33], and SCUT-SFVD [34]. One common characteristic of these datasets is that the biometric samples are single-finger ones. In contrast, FV samples in the CandyFV dataset show four fingers together, which enables finger-fusion for robust FVR.

Early research in vascular biometrics relied on various hand-crafted features such as Repeated Line-Tracking (RLT) [35], maximum curvature (MC) [36], wide-line detection (WLD) [37]. These algorithms extract binary pixel-maps representing the vein-network in the biometric sample allowing to compare them.

Frequency-domain methods for FVR has also been proposed. Yang et al. [38] have used a bank of Gabor filters to enhance veins at different scales and then construct a set of FVCodes that are compared using a Cosine-similarity function. This method is claimed to perform better than Miura’s MC features [36]. These results, however, have been estimated over a proprietary, unpublished dataset. More recently, Kovač and Marák [39] have used Gabor filters to detect feature-points in vein-images. Speeded-up robust features (SURF) [40] are used to generate feature-descriptors for the selected feature-points, to construct biometric templates achieving an FVR accuracy of 99.94% on SDUMLA-HMT.

As a recent review [41] shows, several deep learning approaches for FVR have been proposed. Unlike for face-biometrics, publicly available FV datasets are not large enough to train a convolutional neural network (CNN) from scratch. Up to now, deep-learning based FVR approaches have adapted pre-trained CNNs through transfer-learning on FV datasets to construct feature-extractors. Besides actual FVR, deep-learning based methods have also been used for other purposes such as vein-segmentation, encryption, as well as vein-enhancement [42,43] (see  [41]). We note here the work of Bros et al. [42] proposing a Residual Convolutional Autoencoder (RCAE) for vein-enhancement that reduces the classification error on the UTVFP dataset from 2.1% to 1%. In the present work, we have used this RCAE in our processing pipeline.

3. Hardware Design

3.1. The Sweet Sensor Platform

The main goal of this work is to test several sensors and technologies that can allow contactless VR and PAD with high accuracy. In order to mitigate the difficulties associated with contactless reflective acquisition, we follow a modular and extensible approach where several cameras, sensors, illumination devices and modalities are tested for performance. We chose to use focus our attention on sensors that are sufficiently affordable for consumer products, for instance by using small camera modules and cheap S-Mount lenses. Moreover, we hope that such new sensing technologies will be useful in other domains of Science such as medical research. For the present article, we focus our attention on the following acquisition modalities:  

• NIR HD camera pair, with multi-spectral illumination at 850 nm and 950 nm, for vein recognition (VR).
• Color HD camera with white illumination, used for surface features and PAD.
• Stereo Vision (SV) depth measurement using the NIR camera pair and laser dot projectors.
• Photometric Stereo (PS) to obtain fine grained depth resolution and texture from a set of frames illuminated from different angles.

For this platform, we made the choice of using small camera sensors and miniaturized optics to have an image quality more comparable to consumer devices but also allowing us to have a small stereoscopic baseline adapted to short range depth sensing. This choice dictates the use of small embedded computer, integrated in the platform, that can interface with the low-level MIPI-CSI data link of these camera modules. We selected a Jetson TX2 for this purpose, which also opens interesting possibilities for studying low power embedded algorithms.

A schematic diagram depicting the connection of the different subsystems of the device is shown in (Figure 2b). The sweet sensor platform aims at integrating the various elements, cameras, illumination, computer and electronics in a small footprint, $21\times 21\times 21$ cm, to allow simple operation for data capture. It is completed by a screen, a keyboard and a mouse and is used like a regular computer. Cameras and optical components are mounted on an optical breadboard to allow re-positioning and extension. The enclosure (Figure 3) is made of two aluminum plates connected by four 6 mm stainless rods along which components can slide.

3.2. Camera and Optics

For this platform we selected a Sony IMX296 CMOS sensor board a sensor with good sensitivity in the NIR domain capable of capturing 950 nm light. This sensor has a global shutter allowing for accurate synchronization required by SV. It communicates with the host via a MIPI-CSI. It provides $1440\times 1080$ pixels (1.58 M pixels) with 10-bit resolution, is capable of 60 frames per second (FPS), and with a acceptable sensitivity in the NIR range provide it is complemented with 750 nm low-pass filters. It has 24 dB of analog gain and 24 dB of digital gain, the pixels are $3.4\mu \mathrm{m}\times 3.4\mu \mathrm{m}$ for a total active area of $4.9\times 3.7$ mm, i.e., a 6.3 mm diagonal or $1/2.9$ type sensor.

The sensor size constraints the focal length f of the lens as the FoV should be large enough to capture an hand at a relatively short distance and wider FoV leads to higher lens distortion. We choose to use a f = 4 mm lens which give horizontal, vertical and diagonal angles of view of $49.{64}^{\circ }$, $62.{97}^{\circ }$ and $75.{01}^{\circ }$ respectively. At a working distance of 120 mm, the horizontal, vertical and diagonal FoV in a first order approximation is 111 mm, 147 mm, and 184 mm, respectively. Sensors are in portrait orientation.

The use of S-mount M12 lenses constrains the use of fixed aperture optics. On one hand a bigger aperture gives an advantage in term of collected light, on the other hand bigger apertures reduce then usable Depth of Field (DoF). This last quantity can be computed from the formulae:

$$\begin{array}{cc}\hfill H& ={\displaystyle \frac{f^2}{Nc}}+f\hfill \\ \hfill D_n& ={\displaystyle \frac{s\left(H-f\right)}{H+s-2f}}\hfill \\ \hfill D_f& ={\displaystyle \frac{s\left(H-f\right)}{H-s}},\hfill \end{array}$$

(1)

where H is the hyper-focal distance, f = 4 mm is the focal length, $N=f/2.5$ the f-number of the lens, $c=4\mu \mathrm{m}$ the circle of confusion, s the focussed distance and $D_n$ and $D_f$ the near and far limits. With a focussed distance s = 120 mm, we get H = 1.6 m, $D_n$ = 112 mm, $D_f$ = 129 mm and, $\Delta =D_f-D_n=17\mathrm{mm}$.

The DoF $\Delta$ of only 17 mm only correspond to an area where sharpness is maximal, the final usable range is bigger in practice. A smaller f-number of $f/5$ would lead to a double DoF ($\Delta$ = 35.5 mm) range at the expense of a four time lower collected light.

3.3. Jetson Acquisition and Processing Platform

The choice of MIPI CSI camera modules severely constrains the available platforms that can interface several of them. While Field Programmable Gate Array (FPGA) are ideal for these purposes, we instead chose to use an NVIDIA Jetson TX2 that can acquire up to six 2-lanes camera modules. In addition of being much simpler to setup than an FPGA, these System-on-Module (SoM) are based on a multi-core ARM architecture coupled with a relatively powerful GPUs allowing the use of ML frameworks such as PyTorch (v1.8+). For this work a Jetson TX2 Developer Kit (Tegra X2 Series), with and Auvidea J20 camera (v1.0) expansion card, providing a dual-core NVIDIA Denver 2 64-Bit CPU, 4 ARM Cortex-A57 MPCore low power cores, an NVIDIA Pascal GPU with 256 CUDA cores, 8 GB of shared RAM and 32 GB of eMMC flash memory that is extended with a 512 GB external SSD.

3.4. Illumination, Lasers and Controller

For LED illumination we use a modular approach first developed in  [20] using 4 banks of 4 interchangeable modules (Figure 3), each with up to 16 surface mounted LEDs. Each of the 256 LED can be addressed individually in both current, up to 57 mA per device, and Pulse Width Modulation (PWM). These modules are based on the PCA9745B chip thus requiring no LEDs resistor, are daisy chained and controlled by a serial SPI signal. The LED are powered by a dedicated and robust 5 V DCDC Power Supply Unit (PSU) module delivering up to 8 A to the illumination system. Switching all 256 LEDs takes approximately 1 ms. For legal reasons, we cannot release this simple design.

Since commercial laser drivers are bulky and expensive, we developed our own custom laser driver boards, with projector wavelength of 850 nm, based on the IC-NZN chip. These are capable of driving diode lasers in both optical output power (APC) or constant current (ACC) modes, switching up to 155 MHz with an external signal, interface with most laser diodes types (P,N,M) and provide a convenient PMOD interface. This design is released under an open source license alongside this paper.

The sweet platform utilizes a three-camera setup based on Sony IMX296 sensors, comprising two monochrome/NIR cameras and one RGB camera. The RGB camera captures high-resolution images at $1440\times 1080$ resolution with 10-bit depth. The monochrome/NIR cameras form a stereo-capable pair, also offering high-resolution imaging at $1440\times 1080$ with 10-bit depth, enabling enhanced depth perception and low-light performance.

In order to control camera triggering in real-time, LEDs dimming and lasers switching, an additional custom embedded controller board is used. This board is based on a Teensy 4.1 module and will be described in a subsequent paper.

3.5. Software

The Jetson TX2 runs a Ubuntu 20.04 AArch64 operating system with JetPack 5.1 SDK, the kernel is patched with MIPI CSI drivers from the camera manufacturer and loads a custom device tree. The cameras frames are captured using a custom C++ Python (v3.8) interface that access the low-level V4L2 API. A acquisition and visualization GUI is built upon the PyQTGraph (v0.12) library, data storage and processing is based on a custom library that wraps h5py (HDF structure) and OpenCV, respectively. Parts of this software suite are released under an open source license.

3.6. Lasers and Eye Safety Considerations

Working with lasers, especially in a data collection environment, requires care and caution. In particular, the use of diode lasers in the 850 nm range are a safety concern as these wavelengths can reach the cornea and there is no blinking reflex as the light is mostly invisible. We use a pair of Digigram Technology PPR-CEE850-H68V53-30k laser dots projectors that projects 30 k points with a FoV of $67.7\times 53.4$ degrees. These diode lasers have a rated output power of 200 mW and are operated at 200 mA yielding effective optical power of 200 mW per projector, 100 mW after the grating. These dots projectors use a frontal grating positioned approximately 10 mm from the diode and generate a bundle of 30 k laser points, the power per ray is of the order of 3 $\mu$W and therefore too small to cause any eye damage by itself. The center zero-order mode however is more powerful, the datasheet claims that is below 0.2% of the total power, i.e., 0.4 mW. We measured a slightly higher value of 0.5 mW for the center mode, including neighboring points, using a Thorlabs PM160 Wireless Power Meter (2022 make). This implies a Class 2 standard which is not be a problem for eye safety, especially since the lasers are activated for a 3 ms time period, five times per capture. Moreover, we implemented additional safety measures in order to protect the subjects present for data collection, by closely monitoring laser power draw, by orienting the lasers so they are not directed in the subject and operator eyes, by taking regular measurement of the optical output and by training the operator to the specifics of the system.

4. Calibration, Capture, Pre-Processing and Stereo Reconstruction

In order to test which technologies perform the best, the sweet sensor package has multiple cameras and illumination sources. These distinct cameras, functioning as separate channels, enable the capture of different modalities, such as fingers (Figure 4) and wrist (Figure 5). To obtain good quality data, the sensors, illumination and pre-processing pipelines should be precisely calibrated, as described in this section.

4.1. Illumination and Light Field Calibration

The LEDs banks can be adjusted linearly along the z-axis and turned around the y-axis. Moreover, since the LEDs are individually addressable so the light intensity can be adjusted along the y-axis as well. In order to find the best vertical position, angle and individual LED intensity values we use a custom a ray-tracing software to predict the light intensity and perform a manual optimization in situ with a white paper target.

4.2. Camera Gain Calibration and Frame Alignment

When relative LEDs intensities and bank positions are set, the absolute intensity of the illumination, integration time and camera sensor gain should be setup to achieve the best possible performances. At 950 nm both the LEDs and sensors have quite low performance which should be taken into account. The capture frame rate is fixed at 50 Hz which leaves 20 ms per frame to which should be subtracted the time for switching on and off the LEDs banks (2–3 ms) as well as the sensor time to recover the image frame. In practice the maximum integration time is around 5 ms with these parameters. In order to fix the sensor gain and the shutter time we perform a survey of noise vs gain and the optimal parameters we find is low gain (5 dB) and medium integration time (2 ms).

The specific camera sensors we have for this platform have a defect that a non deterministic number synchronization pulses (between 1 and 3 at 50 Hz) are missed at the beginning of the capture. We develop a method consisting of two 10 frames binary patterns we call claps that are recorded at the beginning and at the end of the capture sequence, starting with 10 blank frames. A pre-processing routine then applies a simple pattern matching algorithm to recover the claps position, check that no frame were missed between those and trim away this data.

4.3. Camera Calibration and Sensors Characterization

It is essential for SV 3D reconstruction to have a very accurate characterization of the relative position of the stereo camera pair (extrinsic parameters), as well as a per camera projection matrix and lens distortion coefficients (intrinsic parameters). We perform two dozen of captures covering the whole FoV with a CharuCo target and extract the parameters with OpenCV. Intrinsic camera parameters are approximated with a 5 parameter lens distortion model and a camera matrix, while extrinsic parameters are given by a 3D rotation matrix and translation vector relative to the left camera.

4.4. Stereo Reconstruction and Camera Views Alignment

Reconstruction of the 3D depth map is done from a pair of rectified, undistorted and aligned, left-right image pair. In particular, the calibration algorithm takes care that the rectification parameters, intrinsic and extrinsic, are tuned such that a point in 3D space appear at the same height in the image plane of the two cameras. If a point in 3D space appear at pixel positions x on the left image and $x^{\prime }$ on the right image, the difference is called the disparity as given by,

$$d=x-x^{\prime }={\displaystyle \frac{Bf}{z}},$$

(2)

where B is the baseline distance between the two cameras, f is the focal length and z is the distance of the 3D point to the image plane. In order to find the disparity map, and depth map, a semi-global matching (SGM) algorithm using a mutual information (Ml)-based pixel-wise matching algorithm  [44] implemented in OpenCV (StereoSGBM). With this depth information we can align the RGB image pixel-wise with the reference left rectified view using a simple projection algorithm.

4.5. Photometric Stereo

We use a series of four images, taken with illumination from each corner of the LEDs banks respectively, as alternative way to extract a depth map known as Photometric Stereo (PS). It is possible to estimate the surface normals of the illuminated object using this technique proposed in [45]. The approach solves the least-square solution for the equation $I=L·N$, with I the intensities observed in the images for each pixels, L the normalized vectors for each light sources and N the normal surfaces. Several assumptions are assumed to simplify the problem:

• The light vectors are assumed constant overall the image, such as emitted from infinitely distant isotropic sources.
• The hand skin is considered with a Lambertian reflectance model, without specular reflection.
• The hand surface is assumed smooth.

In addition to this algorithm, we perform a flat-field calibration to compensate the inverse square law of light propagation, by using a flat reference with a constant albedo. This method makes the assumption that the hand is almost planar, its surface not deviating much from that of the reference plane.

5. Vein Recognition Experiments

A biometrics based identity-verification system functions in two phases—enrollment and probe. Both phases operate on biometric templates—a template being a compact representation of a biometric sample. To enroll a new subject in the biometrics verification system, the subject provides a biometric sample. A template, constructed from this sample, is stored in the biometrics system, associated with the subject’s identity. During the probe phase, the subject claims a certain identity, and provides a new biometric sample. The system then compares probe-template, derived from the probe-sample, with the template enrolled for the claimed identity. If the two templates are sufficiently similar (i.e., the match-score is above a predetermined threshold), we consider that the claimed identity has been verified by the system. In this section we describe the template creation process, and then discuss the results of FV-recognition experiments with various protocols (The Python code for finger-vein recognition experiments has been publicly released [11].)

5.1. Finger-Vein Template Creation

Figure 6 shows the flowchart of the FV-template creation process. Each input FV-sample is an image corresponds to a presentation showing all fingers of the presented hand. First, the four fingers—index-, middle-, ring-, and little-finger—are segmented out from the input image. One template is constructed for each finger separately.

In the finger-segmentation step, we extract individual finger sub-images from the input image. We first generate a foreground mask using adaptive thresholding (Otsu’s method [46]) to isolate the hand region, followed by morphological opening to remove noise. The binary image is scanned horizontally to locate the tip of the tallest finger. We then trace the finger’s boundaries row-by-row by identifying edge pixels, stopping when the width exceeds a reasonable threshold. After segmenting one finger, it is removed from the image, and the process is repeated—up to four times—to detect all fingers.

By the nature of the finger-segmentation process, fingers are detected in order of their height in the input image (the finger closest to the top-edge of the image is detected first, followed by the second-tallest, and so on). In other words, the fingers are not necessarily extracted in the order index-to-little or the reverse. In further processing, we use the relative coordinates of the center-of-gravity of each finger-mask to reorder the fingers in a natural order from index to little. However, this finger-reordering procedure can be reliably applied only when all four fingers have been detected. (If, for example, only three fingers have been detected, then we cannot tell whether these are index, middle and ring fingers, or middle, ring, and little fingers.) For this reason, images where all four fingers are not detected, are excluded from further processing.

Next, a normalization step proposed by Huang et al. is applied to each individual finger-image [37]. This step simply rotates the finger-image to align the longitudinal axis of the finger to the vertical axis as best as possible.

The normalized finger-image is passed to the FV-enhancement module. Here we use a pre-trained autoencoder [42] to enhance the vascular structures in the input image. Preliminary experiments showed that FV-enhancement improves the FV recognition accuracy significantly. Hence, we have included the FV-enhancement module in our processing pipeline. A sample result of the vein enhancement process is shown in Figure 7.

FV patterns are compared based on a set of image-features extracted from the two vein-images being compared. In this work we have used the Maximum-Curvature (MC) features [36]. Here, the finger-vein image is scanned line by line in the direction transverse to the length of the finger. In each scan, the pixels of high local curvature (local second derivative of pixel-values) along the line are marked as suitable feature-points. A sample result of the vein MC-feature-extraction process is shown in Figure 7d. The MC feature-map extracted from an input FV-sample image is considered as the biometric template for the sample.

5.2. Finger-Vein Matching

We have used the method proposed by Miura et al. [36] to compare two MC-feature based templates. This method uses cross-correlation (computed in the frequency domain) to find the position of best match of the two input feature-maps. The cross-correlation coefficient at the best-match position is taken as the match-score between the two templates. In Section 5.4 we present experimental FVR results using the following metrics:

• FMR (False Match Rate): The proportion of comparison attempts between samples from different identities that are incorrectly accepted as a match.
• FNMR (False Non-Match Rate): The proportion of comparison attempts between samples from the same identity that are incorrectly rejected as a non-match.
• HTER (Half Total Error Rate): The average of FMR and FNMR, computed as: HTER = 0.5 × (FMR + FNMR).

5.3. Dataset Construction

Using the sweet platform we have collected a dataset for FV recognition experiments. This dataset, named CandyFV, and the results of our FV recognition experiments are discussed in this section.

FV samples from 120 subjects comprise the CandyFV dataset. For this dataset we attempted to have an even distribution of male and female subjects over three age ranges. The demographic distribution of the subjects who provided vein-biometrics samples for the CandyFV dataset is shown in

Table 1. Age and gender distribution of 120 subjects represented in the CandyFV dataset.

Age-Group (in Years)	Male	Female
18–30	22	19
31–50	20	18
51 and above	20	21
Total	62	58

.

The dataset includes five samples for each hand of each subject. The subject presents the hand with four fingers close together (the fingers-closed modality), over the three cameras, at a distance of roughly 10–12 cm from the cameras. Thumbs may be entirely or partially visible in the samples; they are not used in our experiments.

Each sample includes 20 usable image-frames per camera, captured under a variety of illuminations. In particular, each sample yields three images captured under NIR-850 for each of the two (left, right) NIR cameras, and similarly, three images captured under NIR-950 illumination.

5.4. Hand Recognition Experiments

We first discuss the results of several experiments for hand-identity verification based on individual fingers. Following that we present results of hand verification based on multiple fingers, using score-fusion in various ways.

5.4.1. Single Finger Recognition

FV templates have been compared under eight different protocols, described in

Table 2. List of protocols under which finger-comparison experiments have been performed. For each protocol, the number of enrollment images and probe images in the development (dev) set, as well as in the evaluation (eval) set are provided.

Id	Protocol	# Images Dev		# Images Eval
		Enrol	Probe	Enrol	Probe
P1	LH_left_850	159	8064	159	7836
P2	LH_left_950	141	4908	138	4665
P3	LH_right_850	159	7968	156	7791
P4	LH_right_950	147	6372	144	6300
P5	RH_left_850	156	7200	156	6998
P6	RH_left_950	153	6138	136	4644
P7	RH_right_850	156	7116	156	7032
P8	RH_right_950	144	4842	123	3969

. Each protocol name is composed of three items: <Modality>_<Camera>_<NIR>. The Modality may be ‘LH’ (left hand) or ‘RH’ (right hand). The Camera component ‘left’ or ‘right’ indicates the NIR camera from which the template has been derived, and the NIR component may be ‘850’ or ‘950’, indicating the illumination used to capture the image-sample. Thus, with two options for each of three variables, we have eight different experimental protocols.

For these experiments, first we partition the dataset into two subsets, named the development (dev) set and the evaluation (eval) set. The dev set is used for tuning hyper-parameters of the finger-recognition system for the desired performance. The tuned recognition system is then applied to the data corresponding to the eval set, to quantify the performance of the system. The dev and eval sets have been constructed arbitrarily, based on the numerical subject-id assigned to each subject. Data for the first 60 subjects has been assigned to the dev set and data for the remaining subjects has been assigned to the eval set.

From each image captured by the sweet platform, we extract three individual finger-vein images, corresponding to the index-, middle- and ring-finger recorded in the image. Within each set (dev or eval), we have five samples for each user, for each modality. Typically, in each sample, for each camera (left, right), we have three NIR-850 images and three NIR-950 images. That is, in total, for each camera we have 15 NIR-850 images per subject, and similarly 15 NIR-950 images. Thus, for each of 60 subjects in each partition, we have 45 finger-vein images for each camera and each NIR-illumination.

The next step is to construct enrolment-sets and probe-sets. Here, we have arbitrarily selected one sample for each modality of each subject for the enrolment sample. The remaining samples have been designated as probe-samples. For the finger-vein recognition experiments, each enrolled sample is considered a unique identity.

Each probe-sample has been used for four comparisons– one genuine comparison (with the correctly matched identity), and three zero-effort-impostor (ZEI) comparisons (with non-matched identities). In each ZEI comparison, the claimed-identity for a given probe template is selected randomly. For each protocol, the number of enrolled samples and probe-samples in each set (dev or eval) are shown in Table 2. We note two points from the Table:

• The number of enrollment and probe samples for the protocols with NIR-950 illumination are consistently smaller than those for the NIR-850 protocols;
• The numbers of images used for enrolment are not exactly 180 (three fingers for 60 subjects each).

Both observations can be explained by the finger-segmentation results. Images where the finger-segmentation step failed to detect exactly four fingers have been excluded from these experiments. We noted that there were significantly more finger-segmentation errors for images captured with the NIR-950 illumination, than those captured under the NIR-850 illumination.

We estimate the finger-vein recognition rates for the FMR of 0.1%. In this analysis, the score-threshold is selected such that the FMR over the dev set does not exceed the desired FMR. This score-threshold is then applied the dev set and the eval set, to determine the actual FMR and FNMR rates over each dataset.

In

Table 3. Finger-vein recognition performance at FMR of 0.1%. The lowest HTER value is highlighted in bold characters, and corresponds to the RH_right_850 protocol. The FMR, FNMR, and HTER values are expressed as percentages.

Protocol	Dev Set			Eval Set
	FMR	FNMR	HTER	FMR	FNMR	HTER
P1	0.1	3.82	1.96	0.17	7.3	3.74
P2	0.08	8.7	4.39	0.09	10.05	5.07
P3	0.08	5.77	2.93	0.0	6.46	3.23
P4	0.09	7.74	3.91	0.02	9.45	4.74
P5	0.09	0.66	0.38	0.79	0.61	0.7
P6	0.09	0.57	0.33	0.66	0.60	0.63
P7	0.09	0.33	0.21	0.68	0.45	0.57
P8	0.09	1.39	0.74	0.04	2.27	1.15

we summarize the FMR and FNMR achieved for various evaluation protocols, for single finger recognition, for the FMR ceiling of 0.1%. Results show that the FVR performance is significantly better for the right-hand fingers than for the left-hand fingers. We do not have any logical explanation for this phenomenon. This has probably happened for one of two reasons:

• Increased familiarity with the data-capture procedure– subjects were consistently asked to present the left-hand first, therefore, presentations with the right-hand may have been better, or
• Simply due to right-handedness of most subjects, which may lead to smaller variability in right-hand data, compared to left-hand data.

We also note that the recognition-rates achieved for protocols involving 850 nm NIR illumination are usually somewhat better than the corresponding (i.e., same hand, same camera) protocols involving 950 nm illumination. This result is counter-intuitive. In theory, we expect 950 nm illumination to provide better results than 850 nm, because 950 nm NIR penetrates the soft-tissue of the fingers to a deeper extent than 850 nm NIR. Also, 850 nm NIR tends to produce more speckle noise on the skin-surface. On the other hand, much more power is needed for the 950 nm illumination. Our conjecture is that the sweet platform might not have sufficient power for the 950 nm illumination to provide the expected results.

5.4.2. Hand-Recognition Based on Finger-Score Fusion

In the experiments discussed so far, each finger has been considered a unique identity, that is, the identity of a subject is verified based on only a single finger at a time. Next, we consider each hand of a subject as a unique identity, and attempt to validate the identity of each hand based on combined recognition of three fingers of the hand– the index-finger, the middle-finger, and the ring-finger. The raw scores used for single-finger recognition can be combined to recognize a hand based on the vascular patterns of the three fingers combined. For each hand, the scores the three fingers obtained in each of the eight protocols (Table 2) are combined. Thus, each hand-probe is represented by a 3-dimensional (3-D) feature-vector.

The general approach to score-fusion adopted in this study is as follows: feature-vectors are constructed for each hand-identity using the individual finger-comparison scores. While constructing these feature-vectors, finger-scores are selected either only from genuine-probes, or only from ZEI-probes. In this way we obtain, for each hand-identity, a set of genuine-probe (match) feature-vectors, and another set of ZEI (non-match) feature-vectors. A two-class classifier is then constructed using the feature-vectors in the dev set. This classifier is used to label the feature-vectors of the eval set, to quantify the hand-recognition performance of the system. In this study, we have used Support Vector Machines (SVM) with RBF (radial basis function) kernels, for the score-fusion experiments.

In each protocol in Table 2, we fuse the FVR-scores of the three fingers of the hand corresponding to the protocol. Thus, all probe feature-vectors used in a given experiment represent information from the same hand, captured by the same NIR camera, under the same NIR illumination. The results of FVR-score fusion within each FVR protocol are shown in

Table 4. Results of score-fusion of three fingers of a hand within the same protocol. The best performance, printed in bold, corresponds to the protocol ‘RH_right_850’. The FMR, FNMR, and HTER values are expressed as percentages.

Protocol	Dev Set			Eval Set
	FMR	FNMR	HTER	FMR	FNMR	HTER
P1	0.1	2.68	1.39	0.20	4.29	2.25
P2	0.08	4.2	2.14	0.54	3.57	2.06
P3	0.05	2.71	1.38	0.05	3.93	1.99
P4	0.06	3.04	1.55	0.6	2.67	1.66
P5	0.06	0.0	0.03	0.46	0.0	0.23
P6	0.07	0.0	0.03	0.45	0.0	0.23
P7	0.06	0.0	0.03	0.11	0.0	0.06
P8	0.08	0.46	0.27	0.0	1.51	0.76

. These numbers quantify the performance of the score-fusion system when the FMR over the dev set is limited to 0.1%. That is, the classification-score threshold is selected in such a way that the FMR over the dev set does not exceed 0.1%. This classification-score-threshold is then applied to the classification-scores generated for the eval set, to estimate the FMR and FNMR over the eval set. It may be noted that for the fusion experiments, the score-thresholds are not the raw finger-scores, but the scores generated by the SVM classifier for each probe. To avoid confusion, in these fusion experiments we refer to the threshold applied to the output of the 2-class classifier as the classification-score-threshold.

First, we note that FVR-score fusion improves the hand-recognition performance compared to single finger FVR. For the left hand, the single FVR error-rates (HTER in Table 3), range from 3.5% to 5% for each of the individual fingers. Finger-fusion reduces the left-hand recognition error-rates to about 2% or lower in all four left-hand protocols. For the right-hand, single-finger FVR performance is already very high (FVR in protocols P5–P8 in Table 3). Multi-finger FVR performance for the right hand still reduces the classification error. The best performance, a HTER of 0.06% for the ‘RH_right_850’, is a 10-fold improvement over single-finger FVR in the same protocol. Figure 8 presents a comparative overview of both experimental cases across all evaluated protocols in terms of HTER on the eval set.

6. Concluding Remarks

In this paper we have described the design of a contactless vascular biometrics platform named sweet, that has been developed for the purpose of testing of various technologies and methods for contactless vascular biometrics. With this platform we can record hand vascular images in NIR wavelengths (850 and 950 nm) and skin surface in in color RGB. With a pair of NIR cameras and by varying illumination incidence angle we can also extract precise and detailed depth and surface-normal maps using SV and PS, respectively.

In this work we have focussed on finger-vein (FV) biometrics using this platform. Unlike existing FV sensors that capture vascular data from only one finger at a time, the sweet platform is designed to image four fingers of a hand simultaneously. This enables us to implement multi-finger vein recognition, which is not possible using previous FV devices.

We have collected a large dataset of vascular biometrics samples from 120 subjects. The FV data collected from these subjects has been curated into a dataset named CandyFV. We have also reported FV recognition results based on this dataset.

Our platform has been designed in a modular fashion and allows for a large range of improvements and extensions. In the future we will extend this platform in several ways to improve its acquisition performances, for instance by adding side facing cameras, adding sensors with better performance in the NIR range and by experimenting with illumination, for example using polarized light. We also want to test other technologies, such as Single Pixel Laser Detectors (SPLD) to add capabilities in the SWIR range. We will also improve the pre-processing pipeline with channel fusion such as SV with PS fusion. Moreover, the fast acquisition frame-rate facilitates even more complex algorithms based on image sequence analysis, which we will explore in the future.

The CandyFV dataset offers a large scope for further research: (1) Fusion: Up to now we have only explored score-based fusion. This strategy worked well for combining multiple fingers, but did not lead to further performance-improvement when applied to camera fusion or NIR-wavelength fusion. Next we aim to study data-fusion and feature-fusion methods for combining information from the two NIR cameras, as well as from the two NIR illumination wavelengths; and (2) End-to-end processing: Our initial experiments with CNN based end-to-end FV recognition have not shown results comparable to those presented in this report. This is why end-to-end FV recognition has not been discussed here. In future work we plan to explore this direction more rigorously.