Biometrics are a promising tool that can replace or supplement existing knowledge-based or possession-based authentication/identification systems. Biometrics have recently served as one of the key user authentication and/or identification methods as they provide strong security as well as convenience [1
]. Various biometric technologies have so far leveraged human biological characteristics including fingerprint [6
], face [7
], iris [8
], hand vein [9
], electroencephalogram (EEG) [11
], electrocardiogram (ECG) [13
], as well as a recently explored multispectral skin photomatrix (MSP) [16
To realize practical biometric systems, it is important to satisfy key criteria such as collectability, performance, and circumvention [1
]. A number of studies have proposed solutions to meet the criteria: wearable biometric systems can enhance collectability [17
], multimodal biometrics with advanced machine learning algorithm can deliver high performance [5
], and spoof-resistant schemes for biometrics can overcome circumvention [21
]. However, few studies have addressed all of them at once. For instance, Aronowitz et al. proposed multimodal biometrics for mobile devices using voice, face, and chirography with 0.1% equal error rate (
) in 100 subjects [22
]. However, these biometrics are relatively prone to spoofing via remote access due to the lack of contact-based sensing. Gasti et al. developed an authentication technique for outsourced smartphones [23
]. However, multimodal biometrics with face and voice used in this study are easier to compromise than other contact-based biometrics. In contrast, there are biometric signals potentially useful for wearable devices with spoofing-resistance such as wearable ECG [24
], in-ear EEG [12
], wearable MSP [16
], and wearable vein sensor [26
In this study, we propose a multimodal biometrics using our in-house wearable band integrated with MSP and ECG sensors. The proposed biometric system meets all the requirements for practical biometrics, including collectability by a single wearable device for multiple biometric sensors, performance by multimodal biometrics using MSP and ECG, and spoof-resistance by contact-based sensors for MSP and ECG. Since these sensors are cost-effective, multimodal biometrics with MSP and ECG have an advantage over other multimodal biometrics including iris or face that use relatively expensive image sensors for large-scale deployment [5
]. In addition, the straightforward signal processing of MSP and ECG signals requires less computational resources than other biometrics relying on intensive image processing. In fact, all the algorithms developed in this study can be used in low cost wearable devices with limited power computation, memory, and battery capacity. In this regard, we opt to adopt simple authentication algorithms for ECG [14
], and MSP [16
], as well as their fusion.
describes background information on ECG and MSP biometrics and multimodal biometric algorithms. Section 3
illustrates our integrated ECG and MSP wearable band with signal processing methods and user template guided filtering/normalization scheme for multimodal biometrics. Section 4
presents human experiments and authentication results. Section 5
and Section 6
finalize the article with discussion and conclusion.
4. Experimental Results
In this section, we elaborate on a personal authentication experiment to acquire ECG/MSP data using our in-house wearable wrist band, data processing, and experimental results.
4.1. Data Acquisition
Data acquisition was performed on single bespoke wearable devices, denoted hereafter as A, B, C and D. Devices A and B were the WSB incorporating both the ECG and MSP modules (see Section 3.3
and Figure 11
C), whereas devices C and D were separate MSP sensors including only the MSP module (see Section 3.3
and Figure 8
). Devices A and C implemented Red/IR MSP while devices B and D implemented Yellow/IR MSP. For ECG sensing, devices A and B featured three sensor electrodes, two on the inner side of the strap that made continuous contact with the skin of the wrist whilst the device was worn (see Figure 11
C) and one on the outer side of the strap (see Figure 11
A). During ECG acquisition, participants must touch the outward facing sensor with their index finger. This enabled the WSB to measure a potential difference between the wrist wearing the device and the index finger on the other hand, as shown in Figure 11
B. The ECG sampling rate was 250 Hz and the ECG signal was filtered by a low pass filter with a cut off frequency of 40 Hz. MSP data acquisition was achieved by the sensor arrays in all devices, as shown in Figure 8
and Figure 11
C. In all devices, the MSP sensor arrays were integrated into the wrist band such that it was in close proximity to the skin.
The ECG/MSP data for 150 participants were acquired from this study. All subjects in this study gave their informed written consent prior to participation in the study. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Institutional Review Board of the Ulsan National Institute of Science and Technology (UNISTIRB-16-01-G).
The study procedure for each participant was as follows. Firstly, the participant donned device A (Red/IR mode) on their left wrist. Secondly, the device was adjusted by the experimenter to ensure that the skin of the wrist made good contact with both the ECG electrodes and the MSP matrix. Thirdly, data capture took place in the following sequence:
participants placed their right index finger on the outer facing ECG electrode,
the ECG data stream was acquired for 30 s,
participants removed their right index finger from the ECG electrode,
participants then donned device C on their right wrist; MSP data from both devices A and C were simultaneously acquired from five runs of four cycles in the Red/IR mode (for approximately 20 s),
participants removed devices A and C (Red/IR mode),
participants donned devices B and D (Yellow/IR mode),
MSP data from both these devices were acquired with five runs of four cycles in the Yellow/IR mode (for approximately 20 s),
Finally, participants removed the devices and data capture was finished.
4.2. Data Pre-Processing and Evaluation Criteria
A minute-long ECG data was divided into multiple ECG single through the following procedure: (1) baseline correction using a band-pass filtering, (2) R-peak detection using the Pan-Tompkins algorithm [53
], and (3) extraction of individual P-QRS-T fragments with the length of 160 samples (0.64 s), which were in between −67 samples before and +92 samples after the R-peak. We extracted 36 pulses from the ECG data and repetitively averaged two successive pulses, generating 18 averaged pulses. These averaged pulses were grouped into six records each containing three successive averaged pulses. Thus, the structure of the final ECG data was 150 (number of subjects) × 6 (number of records per subject) × 3 (number of pulses per each record) × 160 (number of ECG samples per pulse).
A single cycle of MSP data acquisition generated a 32-D red feature vector plus a 32-D IR feature vector from the Red/IR mode or a 32-D yellow feature vector plus a 32-D IR vector from the Yellow/IR mode, respectively, and a total of 20 cycles were operated for each mode. Then, we concatenated all the feature vectors per cycle from each mode into a single 128-D feature vector (consisting of 32 Red, 32 Yellow and 64 IR light intensities). From these 20 feature vectors, we randomly selected 18 and grouped them into six records each containing three feature vectors, in order to be consistent with the ECG case. Thus, the structure of the final MSP data was formed as 150 (number of subjects)× 6 (number of records per subject)× 3 (number of vectors per record)× 128 (MSP features per vector). Taken together, six records of ECG and MSP were constructed with three ECG pulses and three MSP feature vectors per record for each subject. In all the authentication procedures of this study, an enrolled template was constructed by averaging six feature vectors (i.e., 6 ECG pulses) from two records (i.e.,
, in Equation (2
)), and an authentication input was constructed by averaging three vectors (i.e., three pulses) from one record (i.e.,
, in Equation (2
For each subject, we chose two out of six records for the enrollment to form a personal template and conducted the authentication test on the remaining four records, together with 596 records of others (four records of 149 subjects). This test was repeated 15 times: for all possible combinations of two out of six records. Then, we repeated the same authentication procedure for all the subjects.
Our proposed authentication method was based on the Euclidean distance between the enrolled biometric template and the testing biometric data for each biometric modality as described in Equation (2
). All the Euclidean distances for ECG and MSP were normalized such that the maximum distance in ECG and the maximum distance in MSP were equal to each other. This procedure enabled us to use the same threshold for both ECG and MSP biometrics and to simplify their fusion.
Several performance evaluation criteria were used for the proposed ECG/MSP multimodal biometrics. (False Acceptance Rate) is the ratio of the number of times when an actual imposter was predicted as genuine over the number of all the actual imposter cases (). (False Rejection Rate) is the ratio of the number of times when an actual genuine was predicted as an imposter over the number of all the actual genuine cases (). was obtained by finding a threshold where . was the detection probability at . As a fixed threshold, determines for the same threshold level, the higher detection probability a biometric method yields, the better performance it achieves. We also used the for comparative performance evaluation under , indicating that the lower reject ratio represents the better performance.
4.3. Proposed Methods Description and Authentication Results
shows the performance evaluation result of the proposed multimodal biometric authentication using the ECG + MSP integrated in WSB. It also shows the performance result using ECG in the WSB and the separate MSP sensor with better contact. Note that applying the majority voting (MV) method for two modalities can be conducted in two different ways: in one way, final acceptance is decided when both modality classifiers accept (AND), and, in the other way, final acceptance is decided when at least one modality classifier accepts (OR). Note that the same distinction algorithm was used for both ECG and MSP signals. In addition, this algorithm was applied to both cases when two signals were measured within a single integrated wristband (ECG + MSP integrated in WSB) or when they were measured separately (ECG + separate MSP sensor).
For both cases of different MSP setups, our multimodal biometrics with the MV based on OR fusion yielded superior performance to the MV based on AND fusion:
using the WSB integrated with ECG and MSP and the combination of ECG and a separate MSP sensor, respectively.
of 0.1% is comparable to the state-of-the-art performance of recent multimodal biometrics [22
]. In addition, the combination of ECG with a separate MSP sensor (ECG w/separate MSP sensor in Table 1
) for better contact remarkably improved
to be as low as 0.2%. Moreover, our proposed method has advantages such as good circumvention property (stealing ECG or MSP would be more difficult than face or voice information in [22
We also examined the effect of GF on performance by analyzing authentication outcomes: without applying guided filter (no GF), partially applying guided filter on ECG signal only (GF
), and applying guided filter on both modalities (ECG and MSP, GF
). By applying GF to each modality, the performance in most cases have been enhanced. Specifically, under an extremely difficult situation (i.e.,
), GF showed a remarkable boost in performance (see Table 1
, MV(OR) result). Figure 12
illustrates changes in
with various thresholds. Note that
are non-decreasing and non-increasing, respectively, for an increasing threshold.
The proposed wearable wrist band integrated with MSP and ECG sensors enabled multimodal biometrics in a single portable device and demonstrated a feasibility to provide strong security with advantages in collectability (single wearable wrist band with multimodal biometric sensors), performance (multimodal fusion algorithm with MSP and ECG), and circumvention (MSP and ECG data that are not easy to steal remotely).
In the present study, our proposed wearable wrist band with two sets of sensors collects biometric signals and then the authentication process is conducted separately in a computer. Even though this scheme is still useful as it is, it will be more widely applicable and more secure if the authentication process can be performed inside the band. It is potentially possible in our current algorithm due to its simplicity. Our proposed multimodal biometrics method exploits a simple Euclidean distance measure for individual biometrics as well as a simple fusion scheme based on majority voting. These algorithms only require lightweight computation so that they are suitable for wearable devices with limited computation power. Even though there are many computationally powerful wearable devices these days, heavy computation still requires high energy consumption. Thus, our proposed algorithms have an advantage over other algorithms in terms of computational loads.
For the development of multimodal biometrics in the wearable band, we conjecture that the wearable band should allow limited access to others’ biometric data. Most state-of-the-art biometrics methods yield excellent results by exploiting high-dimensional feature spaces and sophisticated decision rules based on large amounts of training data (e.g., see [13
]). However, our proposed method only requires the computation of a few parameters (distances
and a threshold
). Our preliminary study (not shown here) suggested that weights in multimodal fusion could be obtained from a small number of sample data so that it may be possible to fix these values. In that case, no additional biometric information will be required for a stand-alone wearable band equipped with sensors and authentication processing. Further investigation will be necessary to confirm this conjecture.
Odinaka et al. showed that ECG biometrics could yield up to 0.03% EER, but with 64 ECG pulses for the user template generation and another 64 ECG pulses for authentication which were acquired on the same day per subject using medical grade ECG acquisition systems [13
]. They also showed that the same authentication method could yield up to 0.38% EER with 32 pulses. In ECG, using more pulses implies longer acquisition time for authentication-e.g., acquiring 64 pulses would take approximately 1 min. In this study, only averaged three ECG pulses were used for fast response authentication and by fusing it with MSP efficiently, the proposed method yielded up to 0.1% EER, which is much better than ECG biometrics using 32 pulses and other subjects’ data for sophisticated classifer training. Our proposed methods have advantages of both high performance and practical response time over other state-of-the-art methods.
Even though many promising results were presented in this article, there are also limitations. All the biometric data were acquired on the same day, so a possible biometric signal variation over time in individuals was not tested. However, there have been many works to deal with ECG signal variation over time for authentication. For instance, Chun proposed methods to deal with short-term and long-term ECG signal variations for high performance ECG based authentication [15
]. Incorporating signal variation models in the proposed methods and validating them with extensive biometric data sets can be an important next step of the work.
Furthermore, we will pursue generalizing proposed wearable wrist band-type multimodal authentication system for practical usage. Also, in this study, we were only concerned with normal sinus ECG rhythms to test our integrated system for a general population. However, it would be also important to consider applying our system to specific cases such as the users with arrhythmia or vertical heart. In addition, there is a possibility for changes in ECG signals due to various status of subjects such as heart rate or orientation of hearts. In the present study, we controlled the orientation of the heart by letting the subjects stand still during measurements. In addition, we also controlled the heart rate of the subjects as there are many studies showing that ECG signal changes, especially T-wave changes, due to changes in heart rates (e.g., running or standing steady) [15
]. Since our proposed algorithm leverages the shape of ECG signal itself, it may need to be tested our algorithm for the identification of those cases as well in the follow-up studies.
In this article, we performed authentication in two ways of measuring MSP signals, one fully integrated in the WSB and the other built separately to enhance contact with the skin. Since the MSP signals can be influenced by ambient lights, it is emphasized in this study that the photomatrix system was firmly attached to the skin to prevent influences of ambient lights. To achieve this, we fabricated MSP modules with various curvatures and tested a fit of each curvature to the wrist surface. It allowed us to find the best fit between the curvature and the wrist surface so that the MSP signals were hardly affected by external lights. In addition, we asked the user not to move during the measurements to minimize motion artifacts. In doing so, we have not observed significant influences of potential artifacts. Otherwise, it is also possible for MSP signals to be influenced by other motion artifacts generated by placing a finger on the device when the ECG is measured. To avoid this, we can start to measure the data after the finger is positioned and make sure that users do not move after touch, so that ECG and MSP will not be affected by motion artifacts. However, in the present study, we have adopted the sequential measurement method instead of the simultaneous measurement in consideration of a possibility that the pressure generated from the finger may affect the MSP signals.
On the other hand, there might be an effect if the user puts the MSP sensor in another position. When photons emitted from the LED meet components in the skin tissue, they are absorbed or diffuse-reflected. A unique MSP pattern is generated from the behaviors of these photons. If the user places the device at another position, the internal structure of the skin (bone, ligament, collagen, etc.) that the photons meet in the skin will change. As such, the angle and depth of diffuse-reflectance become different, which can result in changes in the MSP pattern. This would lead to an increase in by more failure to identify the user.
The results showed substantial improvement of performance using the separate MSP sensor, demonstrating the importance of reliable contact between the MSP sensors and the wrist skin. From this, our follow-up research is developing an advanced version of the WSB with the new structure that ensures firm contact of the MSP sensor. Lastly, we additionally examined the effect of the number of users on multimodal authentication performance. We found that the performance of authentication using ECG + MSP integrated in WSB started to decrease in terms of score when 50 or more users were tested. In contrast, the performance using the ECG + separate MSP sensor did not degrade much as the number of users increased. To generalize this effect, we will test our methods with a much larger number of subjects.