Mobile Touchless Fingerprint Recognition: Implementation, Performance and Usability Aspects

This work presents an automated touchless fingerprint recognition system for smartphones. We provide a comprehensive description of the entire recognition pipeline and discuss important requirements for a fully automated capturing system. Also, our implementation is made publicly available for research purposes. During a database acquisition, a total number of 1,360 touchless and touch-based samples of 29 subjects are captured in two different environmental situations. Experiments on the acquired database show a comparable performance of our touchless scheme and the touch-based baseline scheme under constrained environmental influences. A comparative usability study on both capturing device types indicates that the majority of subjects prefer the touchless capturing method. Based on our experimental results we analyze the impact of the current COVID-19 pandemic on fingerprint recognition systems. Finally, implementation aspects of touchless fingerprint recognition are summarized.


Introduction
Fingerprints are one of the most important biometric characteristic due to their known uniqueness and persistence properties.Fingerprint recognition systems are not only used worldwide by law enforcement and forensic agencies, they are also deployed in the mobile devices as well as in nation-wide applications.The vast majority of fingerprint capturing schemes requires contact between the finger and the capturing device's surface.These systems suffer from distinct problems, e.g.low contrast caused by dirt or humidity on the capturing device plate or latent fingerprints of previous users (ghost fingerprints).Especially in multi-user applications hygienic concerns lower the acceptability of touch-based fingerprint systems and hence, limit their deployment.In a comprehensive study, Okereafor et al. [1] analyze the risk of an infection by touch-based fingerprint recognition schemes and the hygienic concerns of their users.The authors conclude that touch-based fingerprint recognition carries a high risk of an infection if a previous user has contaminated the capturing device surface, e.g. with the SARS-CoV-2 virus.
To tackle these shortcomings of touch-based schemes, touchless fingerprint recognition systems have been researched for more than a decade.Touchless capturing schemes operate without any contact between the finger and the capturing device.Several contributions to the research area paved the way for a practical implementation of touchless capturing schemes.Specialized stationary capturing devices based on multi-camera setups combined with a powerful processing have already been implemented in a practical way [16].However, to the best of the authors' knowledge no approach to mobile touchless recognition scheme based on of-the-shelf components has been documented so far.
In this work, we propose a mobile touchless fingerprint recognition scheme based for smartphones.We provide a fully automated capturing scheme with integrated quality assessment in form of an Android app.All components of the processing pipeline will be made publicly available for research purposes.The rest of the recognition pipeline is composed of open source algorithms for feature extraction and comparison.
To benchmark our proposed recognition pipeline we acquired a database under real life conditions.A number of 29 subjects have been captured by two touchless capturing devices in different environmental situations.Moreover, touch-based samples have been acquired as baseline and to measure the interoperability between both schemes.A usability study, which has been conducted after the capturing, reviews the users' experiences with both capturing device types.This paper details on every implementation step of the processing pipeline and summarizes our database acquisition setup in detail.Further, we evaluate the biometric performance of our acquired database and discusses the outcomes of our usability study.Based on our experimental results we elaborate on the impact of the current COVID-19 pandemic on fingerprint recognition in terms of biometric performance and user acceptance.Further we summarize implementation aspects which we consider as beneficial for mobile touchless fingerprint recognition.
The rest of the paper is structured as follows: Section 2 gives an overview on touchless end-to-end schemes proposed in the scientific literature.In Section 3, the proposed processing pipeline is presented.In Section 4, we describe our experimental setup and provide details about the captured database and the usability study.The results of our experiments are reported in Section 5.The influence of the COVID-19 pandemic on fingerprint recognition is discussed in Section 6. Section 7 discusses implementation aspects.Finally, Section 8 concludes.

Related Work
In this section, we present an overview on touchless fingerprint end-to-end solutions along with reported biometric performance.Table 1 summarizes most relevant related works and their properties.For a comprehensive overview on the topic of touchless fingerprint recognition the reader is referred to [17].
The research on touchless fingerprint recognition has evolved from bulky single finger devices to more convenient multi-finger capturing schemes.First end-to-end approaches with prototypical hardware setups were presented by Hiew et al. [2] and Wang et al. [4].Both works employed huge capturing devices for one single finger acquisition within a hole-like guidance.
For remote user authentication, Piuri et al. [3] and Kumar et al. [5] investigated the use of webcams as fingerprint capturing device.Both schemes showed a very low EER in experimental results.However, the database capturing process was not reported precisely.In addition, the usability and user acceptance of such an approach should be further investigated.
More recent works use smartphones for touchless fingerprint capturing.Here, a finger image is taken by a photo app and manually transferred to a remote device where the processing is performed [10,11].The improvement of the camera and processing power in current smartphones has made it possible to capture multiple fingers in in a single capture attempt and process them on the device.Stein et al. [8] showed that it is feasible for the automated capturing of a single finger image using a smartphone.Carney et al. [12] presented the first four-finger capturing scheme.Weissenfeld et al. [14] proposed a system with a free positioning of four fingers in a mobile prototypical hardware setup.
In summary, it can be observed that the evolution of touchless fingerprint technologies has moved towards outof-the-box device and a more convenient and practically relevant recognition process.Table 1 also indicates that both an accurate recognition and a convenient capturing are hard to achieve.It should be further noted that the level of constraints during the capturing has a major influence on the recognition performance.

Mobile Touchless Recognition Pipeline
An unconstrained and automated touchless fingerprint recognition system usually requires a more elaborated processing compared to touch-based schemes.Figure 1 gives an overview on the key processing steps of the proposed recognition pipeline.Our method features an on-device capturing, pre-processing, and quality assessment whereas the biometric identification workflow is implemented on a back-end system.This section describes each component of the recognition pipeline in detail.
The proposed method combines three implementation aspects seen as beneficial for an efficient and convenient recognition:  • A smartphone as hardware platform which captures the finger image and obtains a fingerprint images from it.
• A free positioning of the fingers without guidelines or framing.
• A fully automated capturing and processing with an integrated quality assessment and user feedback.
The whole capturing and processing pipeline discussed in this section is made publicly available2 .

Capturing
The vast majority of mobile touchless recognition schemes rely on state-of-the-art smartphones as capturing devices.Smartphones offer a high-resolution camera unit, a powerful processor, an integrated user feedback via display and speaker, as well as a mobile internet connection for on-demand comparison against centrally stored databases.
In our case, the capturing as well as the processing is embedded in an Android App.Once the recognition process is started the application analyzes the live-view image and automatically captures a finger image if the quality parameters fit the requirements.During capturing the user is able to see his/her fingers through a live-view on the screen and is able to adjust the finger position.In addition, the capturing progress is displayed.

Segmentation of the Hand Area
Proposed strategies for the segmentation mainly rely on color and contrast.Many works use color models for segmenting the hand color from the background.Here an Otsu's adaptive threshold is preferable over static thresholding.Combinations of different color channels also show superior results compared to schemes based on one channel [18,19,20,21].
Figure 2 presents an overview of the segmentation workflow.We adopt this method and analyze the Cr component of the yCbCr color model and the H component of the HSV color model.As first step we normalize the color channels to the full range of the histogram.Subsequently, the Otsu's threshold determines the local minimum in the histogram curve.A binary mask is created where all pixel values below the threshold are set to black and all pixels above the threshold are set to white.
Additionally, our algorithm analyzes the largest connected components within the segmentation mask.Ideally the segmentation mask should only contain one to four dominant components: from one hand area up to four finger areas, respectively.The method also checks the size and shape of segmented areas.

Rotation Correction, Fingertip Detection and Normalization
The rotation correction transforms every finger image in a way that the final fingerprint image is oriented in an upright position.
Figure 3 presents an overview of the rotation correction, fingertip detection and normalization.Our method features two rotation steps: First a coarse rotation on the full hand and second a fine rotation on the separated finger.A robust separation and identification of the fingers requires that the hand is rotated to an upright position.Here the image border of the binary segmentation mask is analyzed.Many white border pixels indicate that the hand is placed into the sensor area from this particular direction.For this reason, we search for the border area with the most white  pixels and calculate a rotation angle from this coordinate.Figure 4 illustrates this method.
After the coarse rotation the fingertips are separated.To this end, the amount of contours of considerable size is compared to a pre-configured value.If there are fewer contours than expected it is most likely that the finger images contain part of the palm of the hand.In this case pixels are cut out from the bottom of the image and the sample is tested again.In the case of more considerable contours, the finger image is discarded in order to avoid processing wrong finger-IDs.
An upright rotated hand area does not necessarily mean that the fingers are accurately rotated because fingers can be spread.A fine rotation is computed on every finger image to correct such cases.Here, a rotated minimal rectangle is placed around every dominant contour.This minimal rectangle is then rotated into an upright position.
Additionally, the hight of each finger image needs to be reduced to the area which contains the fingerprint impression.Other works have proposed algorithms which search for the first finger knuckle [22,23].We implemented a simpler method which cuts the hight of the finger image to the double of its width.In our use case this method leads to slightly less accurate result but is much more robust against outliers.
Touchless fingerprint images captured in different sessions or processed by different workflows do not necessarily have the same size.The distance between sensor and finger defines the scale of the resulting image.For a minutiaebased comparison it is crucial that both samples have the same size.Moreover, in a touchless-to-touch-based interoperability scenario the sample size has to be aligned to the standardized resolution, e.g.500dpi.Therefore, we normalize the fingerprint image to a width of 300px.This size approximately corresponds to the size touch-based fingerprints captured with 500dpi.
Together with the information which hand is captured, an accurate rotation correction also enables a robust identification of the finger ID, e.g.index, middle, ring or pinky finger.

Fingerprint Processing
The pre-processed fingerprint image is aligned to resemble the impression of a touch-based fingerprint.Figure 5 presents the conversion from a finger image to a touchless fingerprint.We use the Contrast Limited Adaptive Histogram Equalization (CLAHE) on a gray scale converted fingerprint image to emphasize the ridge-line characteristics.
Preliminary experiments showed that the used feature extractor detects many false minutiae at the border region of touchless fingerprint samples.For this reason, we crop approx.15 pixel of the border region.That is, the segmentation mask is dilated in order to reduce the size of the fingerprint image.

Quality Assessment
Quality assessment is a crucial task for touch-based and touchless fingerprint recognition schemes.We distinguish between two types of quality assessment: An integrated plausibility check at certain points of the processing pipeline and a quality assessment on the final sample.
The integrated plausibility check is an essential precondition for a successful completion of an automated recognition scheme.It ensures that only samples which passed the check of a processing stage are handed over to the next stage.In the proposed pre-processing pipeline we implement three plausibility checks: • Segmentation: Analysis of the dominant components in the binary mask.Here the amount of dominant contours as well as their shape, size and position are analyzed.Also, the relative positions to each other are inspected.
• Capturing: Evaluation of the fingerprint sharpness.A Sobel filter evaluates the sharpness of the processed gray scale fingerprint image.A square of 32×32 pixels at the center of the image is considered.A histogram analysis then assesses the sharpness of the image.• Rotation, Cropping: Assessment of the fingerprint size.The size of the fingerprint image after the cropping stage shows if the fingerprint image is of sufficient quality.
The combination of these plausibility checks has shown to be robust and accurate in our processing pipeline.Every sample passing all three checks is considered as a candidate for the final sample.For every finger ID five samples are captured and processed.All five samples are finally assessed by NFIQ2.0 and the sample with the highest quality score is considered as the final sample.An assessment on the applicability of NFIQ2.0 on touchless fingerprint presented is shown in [24].Especially for selecting the best sample from a series of the same subject, NFIQ2.0 is wellsuited.

Feature Extraction and Comparison
As mentioned earlier, the presented touchless fingerprint processing pipeline is designed in a way that obtained fingerprints are compatible with existing touch-based minutiae extractors and comparators.This enables the application of existing feature extraction and comparator modules within the proposed pipeline and facilitates a touchbased-to-touchless fingerprint comparison.Details of the employed feature extractor and comparator are provided in Section 5.

Experimental Setup
To benchmark our implemented app we conducted a data acquisition along with a usability study.Each volunteering subject first participated in a data acquisition session and then was asked to answer a questionnaire.

Database Acquisition
For the capturing of touchless samples two different setups were used: Firstly, a box-setup simulates a predictable dark environment.Nevertheless, the subject was still able to place their fingers freely.Secondly, a tripod-setup simulates a fully free capturing setup where the instructor or the subject holds the capturing device3 .For the touchless database capturing we used two different Smartphones: the Huawei P20 Pro (tripod setup) and the Google Pixel 4 (box setup).The finger images are captured with the highest possible resolution.The flashlight was turned on during the capturing process by default.Also, touch-based samples were captured to compare the results of the proposed setup against an established system.On every capturing device the four inner hand fingers (finger IDs 2 -5 and 7 -10 according to ISO/IEC 19794-4 [26]) were captured.The capturing of with three capturing devices shown in Figure 6 was conducted in two rounds.
To measure the biometric performance of the proposed system, we captured a database of 29 subjects.The age and skin color distribution can be seen in Figure 7. Table 2 summarizes the database capturing method.During the capturing of one subject, Failure-To-Acquire (FTA) errors according to ISO/IEC 19795-1 [27] occurred on both  touchless capturing devices.Interestingly, this was likely caused by the length of the subject's fingernails.For more information the reader is referred to Section 7.6.In total, we captured and processed 1,360 fingerprints.

Usability Study Design
A usability study was conducted with each subject after they had interacted with the capturing devices.Each subject was asked about their individual preferences in terms of hygiene and convenience during the capturing process.Parts of our usability study are based on [28,14].We ensured that the questionnaire is as short and formulated as clearly as possible such that the participants understood all question correctly [29].
The questionnaire is provided as supplemental material.It contains three parts: The first part contains questions about the subject's personal preferences.The Question 1.2b and 1.2c are aligned with Furman et al. [28].Here the different perceptions for personal hygiene before and during the COVID-19 pandemic were asked.The answer options of Question 1.5 were rated by the capture subjects using the Rohrmann scale [30] (strongly disagree, fairly disagree, undecided, fairly agree, strongly agree).The questions were intended to find out the subjects' perception regarding hygienic concerns during the fingerprint capturing process.
The second part of the questionnaire contains questions about the dedicated usability of a capturing device.The same questions were answered by the subject for both devices.This part was designed so that the same questions for both capturing devices were asked separately from each other in blocks.The intention behind this is to conduct comparisons between the different capturing devices.Again, the Rohrman scale was used and sub-questions were arranged randomly.In the last part of the questionnaire, the subjects were asked about the personal preference between both capturing devices.Here, the subjects had to choose one preferred capturing device.

Results
This section presents the biometric performance achieved by the entire recognition pipeline and the outcome of our usability study.

Biometric Performance
In our experiments, we first estimate the distributions of NFIQ2.0 scores for the captured data set.Additionally, the biometric performance is evaluated employing open-source fingerprint recognition systems.The features (minutiae triplets -2-D location and angle) are extracted using a neural network-based approach.In particular, the feature extraction method of Tang et al. [31] is employed.For this feature extractor pre-trained models are made available by the authors.To compare extracted templates, a minutiae pairing and scoring algorithm of the sourceAFIS system of Važan [32] is used 4 .
In a first experiment, we compare the biometric performance on all fingers between the different sub data sets.From the Table 3 we can see that the touchless box setup obtains an Equal Error Rate (EER) of 10.71% which is comparable to the touch-based setup (8.19%).this drop in terms of EER.Here, all three data sets have a comparable average score.In a second experiment, we compute the biometric performance for every finger separately5 .From Table 4 and Figure 9 we can see that on all subsets the performance of the little finger drops compared to the other fingers.On the touch-based sub data set the middle finger has a much lower EER (1.72%) than the rest.This could be because it might be easiest for users to apply the correct pressure to the middle finger.From Figure 10 it is observable that there is only a small drop of NFIQ2.0 quality scores on the little finger.
Further we applied a score level fusion on 4 and 8 fingers.Obtained EERs are summarized in Table 5.As expected,  the fusion improves the EER on all sub data sets.In particular, the fusion of 8 fingers shows a huge performance gain (see Figure 11).The box setup and the touch-based sensor show an EER of 0% which means that matches and nonmatches are completely separated.The open setup also achieves a considerably high performance gain trough the fusion.Here, the inclusion of all fingers makes the process much more robust especially in challenging environmental situations.
In our last experiment we analyze the interoperability between the different subsets of the collected data.Table 6 summarizes the EERs achieved by comparing the samples of different setups and Figure 12  It should be noted, that all data sets, touchless as well as touch-based show a biometric performance which is inferior compared to the state-of-the-art.This is caused mainly by two reasons: On the one hand, all data sets were captured under realistic scenarios.The amount of instructions given to the capturing subject were limited to a minimum.Further, no re-capturing of poor quality samples took place.On the other hand, hygienic measures have an impact on  No. 2.1a: I think the fingerprint scanner was too slow.
No. 2.1b: I found it hard to take the right position for the recording process.
No. 2.1c: I found it hard to keep the position for the recording process.
No. 2.1d: I always knew if the recording process was in progress.
No. 2.1e: I would consider using this capturing device in public places.
No. 2.1f: I found the recording process comfortable.
No. 2.1g: I have hygienic concerns about using the capturing device.biometric performance which is discussed in detail in Section 6.

Usability Study
We present the results of our usability study based on the questionnaire introduced in Section 4.2.The questionnaire was answered by 27 subjects (8 female, 19 male).The subjects were between 22 and 60 years old (average age: 31.22 years, median age: 28 years).The age distribution is presented in Figure 7.The exact result in terms of median and standard deviation is provided as supplemental material.The majority of subjects have used professional fingerprint scanners before this study.A large proportion of the data subjects also use any type of fingerprint capturing device regularly at least once per week, e.g. to unlock mobile devices.
Figure 13 presents the perceptions of the subjects regarding general hygiene.The subjects tend to have general concerns about touching surfaces in public places (Statement 1.5b).Moreover, the majority has personal concerns re-lated to the COVID-19 pandemic (Statement 1.5c).From the small difference in terms of perception before and during the COVID-19 pandemic it can be observed that the pandemic has only a small influence on the general hygienic awareness of the tested subjects.
The usability assessment of the touchless and touchbased capturing devices is presented in Figure 14.In most statements, both capturing devices were rated fairly similar.The touchless capturing device has a slight advantage in terms of capturing speed (Statement 2.1a).The touchbased capturing device tends to be rated better in taking and keeping the capturing position during the whole process (Statements 2.1b and 2.1c).Also, the subjects found it slightly easier to assess if the capturing process is running (Statement 2.1d).Moreover, it can be observed that the subjects highly prefer the comfort of the touchless device (Statement 2.1f).Most notably, the subjects have much less hygienic concerns using the touchless device in public places (Statements 2.1.eand 2.1g).In these cases, a U-Test [33] shows a two-sided significance with a level of α = 5%.Figure 15 illustrates the comparative results.In a direct comparison of the different capturing device types the advantage of hygiene outweighs the disadvantages of hand positioning.The slight majority of subjects would prefer the touchless capturing device over a touch-based one in terms of general usability (Question 3.1).Considering hygienic aspects the vast majority would choose the touchless capturing device over the touch-based one (Question 3.2).This highly correlates to the assessment of hygienic concerns of the statement 1.5c.

Impact of the COVID-19 Pandemic on Fingerprint Recognition
The accuracy of some biometric characteristics may be negatively impacted by the COVID-19 pandemic.The pandemic and its related measures have no direct impact on the operation of fingerprint recognition.Nevertheless, there are important factors that may indirectly reduce the recognition performance and user acceptance of fingerprint recognition.

Impact of Hand Disinfection on Biometric Performance
The biometric performance drops due to dry and worn out fingertips.Olsen et al. [37] showed that the level of moisture has a significant impact on the biometric performance of touch-based fingerprint recognition systems.The authors tested five capturing devices with normal, wet and dry fingers.Especially dry fingers have been shown to be challenging.Also, medical studies have shown that a frequent hand disinfection causes dermatological problems [38,39].The disinfection liquids dry out the skin and cause chaps in the epidermis and dermis.Thus, we can infer that the regular hand disinfection leads to two interconnected problems which reduce the recognition performance: Dry fingers show low contrast during the capturing due to insufficient moisture.In addition, disinfection liquids lead to chaps on the finger surface.Figure 16 shows touch-based fingerprints captured before the COVID-19 pandemic (a) and during the COVID-19 pandemic (b).Both samples were captured from the same subject using the same capturing device.It is observable that the sample (b) exhibits more impairments in the ridgeline pattern compared to sample (a).Moreover, the finger image (c) clearly shows chaps in the finger surface which are likely caused by hygienic measures.The processed touchless sample (d) shows these impairments, too.
Table 7 summarizes the biometric performance achieved on different databases using the proposed recognition pipeline.We can see that the biometric performance on the fingerprint subcorpus of the MCYT bimodal database [34] and the Fingerprint Verification Contest 2006 (FVC06) Table 7: Average NFIQ2.0 scores and biometric performance obtained from touchless and touch-based databases including the fingerprint subcorpus of the MCYT Database [34], the FVC2006 Database [35], the Hong Kong Polytechnic University Contactless 2D to Contact-based 2D Fingerprint Images Database Version 1.0 [36] [35] show a good performance.Moreover, the touchless subset of the Hong Kong Polytechnic University Contactless 2D to Contact-based 2D Fingerprint Images Database Version 1.0 (PolyU) [36] shows an competitive performance of 3.50% EER.Compared to these baselines the performance achieved on our database is inferior which is most likely the impact of the unconstrained acquisition scenario as well as the use of hand disinfection measures.

User Acceptance
Viruses, e.g.SARS-CoV-2, have four main transmission routes: droplet, airborne, direct contact, and indirect contact via surfaces.In the last case an infected individual contaminates a surface by touching it.A susceptible individual who touches the surface afterwards has a high risk of infection via this indirect transmission route.Otter et al. [40] present an overview on the transmission of different viruses (including SARS coronaviruses) via dry surfaces.The authors conclude that SARS coronaviruses can survive for extended periods on surfaces and for this reason form a high risk of infection.
Especially in large scale implementations e.g. the Schengen Entry/Exit System (EES) [41] were many individuals touch the surface of a capturing device, the users are exposed to a major risk of infection.The only way of a safe touch-based fingerprint recognition in such application scenarios is to apply a disinfection of the capturing device after every subject.
Nevertheless, the requirement of touching a surface can lower the user acceptance of touch-based fingerprint recognition.The results of our usability studies in Subsection 5.2 show that the asked individuals are fairly skeptical about touching capturing device surfaces in public places and that (in a direct comparison) they would prefer a touchless capturing device.For this reason the touchless capturing schemes could lead to a higher user acceptance.However, it should be noted that our tested group was very small and user acceptance is dependent to the capturing device design.

Implementation Aspects
This section summarizes aspects which are considered beneficial for a practical implementation.

Four-Finger Capturing
As it has been shown in previous works, our proposed recognition pipeline demonstrates that it is possible to process four fingerprints from a continuous stream of input im-ages.This requires a more elaborated processing but has two major advantages: • Faster and more accurate recognition process: Due to a larger proportion of finger area in the image, focusing algorithms work more precisely.This results in less miss-focusing and segmentation issues.
• Improved biometric performance: The direct capturing of four fingerprints in one single capturing attempt is highly suitable for biometric fusion.As show in Table 5 this lowers the EER without any additional capturing and very little additional processing.
However, a major obstacle for touchless schemes is to capture the thumbs accurately and conveniently.In most environments, the best results are achieved with the innerhand fingers facing upwards.This is ergonomically hard to achieve with thumbs.

Automatic Capturing and On-Device Processing
State-of-the-art smartphones feature powerful processing units which are capable to execute the described processing pipeline in a reasonable amount of time.We have shown that a robust and convenient capturing relies on an automatic capturing with integrated plausibility checks.Also, the amount of data which has to be transferred to a remote recognition workflow is reduced by an on-device processing and the recognition workflow can be based on standard components.
Especially in a biometric authentication scenario, it can be beneficial to integrate the feature extraction and comparison into the mobile device.In this case, an authentication of a previously enrolled subject can be implemented on a stand-alone device.

Environmental Influences
Touchless fingerprint recognition in unconstrained environmental situations may be negatively affected by varying and heterogeneous influences.In our experiments, we showed that our touchless setup performs rather good under a semi-controlled environment.The performance of the same recognition pipeline drastically drops in an uncontrolled environment.Here, it is observable that different stages of the processing pipeline suffer from challenging environments: • Focusing of the hand area needs to be very accurate and fast in order to provide sharp finger images.Here, a focus point which is missed by a few millimeters causes a blurred and unusable image.Figure 17(a, d) illustrate the difference between a sharp finger image and a slightly de-focused image with help of a Sobel filter.Additionally, the focus has to follow the hand movement in order to achieve a continuous stream of sharp images.The focus of our tested devices tend to fail under challenging illuminations which was not the case in the constrained environment.
• Segmentation, rotation and finger separation rely on a binary mask in which the hand area is clearly separated from the background.Figure 17(b, e) show examples of a successful and unsuccessful segmentation.Impurities in the segmentation mask lead to connected areas between the fingertips and artifacts at the border region of the image.This causes inaccurate detection and separation of the fingertips and incorrect rotation results.Because of heterogeneous background this in more often the case in unconstrained setups.
• Finger image enhancement using the CLAHE algorithm normalizes dark and bright areas on the finger image.From Figure 17(c, e) we can see that this also works on samples of high contrast.Nevertheless, the results on challenging images may become more blurry.
The discussed challenges lead to a longer capturing time and for this reason lower the usability and user acceptance.Further, the recognition performance in unconstrained environments is limited.Here, a weighing between usability and performance should be done based on the intended use case of the capturing device.The quality assessments implemented in our scheme detect these circumstances and discard finger images with said shortcomings.More elaborated methods could directly adapt to challenging images, e.g. by changing the focusing method or segmentation scheme.This approach could lead to more robustness and hence improved usability in different environments.

Feature Extraction Strategies
Feature extraction techniques are vital to achieve a high biometric performance.In our experiments, we used an open-source feature extractor which is able to process touch-based and touchless samples.Figure 18 shows an example of this minutiae-based feature extraction and comparison scheme.With this method we were able to test the interoperability between capturing device types.Nevertheless, the overall performance may be improved by more so-phisticated methods, e.g.commercial of-the-shelf-systems like the VeriFinger SDK [42].
Touchless fingerprint images do not correspond to the standardized 500dpi resolution of touch-based capturing devices because of a varying distance between capturing device and finger.Here a feature extractor which is robust against these resolution differences or a dedicated processing step is beneficial.
Also, dedicated touchless feature extraction methods can increase the performance as shown in [11,43].Here the authors are able to tune their feature extractor to their capturing and processing and propose an end-to-end recognition system.

Visual Instruction
According to the presented results in Subsection 5.2, the visual feedback of the touchless capturing device have also been rated to be inferior compared to the touch-based one.
Here the smartphone display is well-suited to show further information about the capturing process.Additionally, an actionable feedback can be given on the positioning of the fingers, as suggested in [12].

Robust Capturing of different Skin Colors and Finger
Characteristics An important implementation aspect of biometric systems is that they must not discriminate certain user groups based on skin color or other characteristics.During our database capturing subjects of different skin color types were successfully captured.Nevertheless, it must be noted that the amount of subjects is too small to make a general statement about the fairness of the presented approach.
As already mentioned in Section 4.1, we observed one single failure to acquire (FTA) during our database acquisition.Most likely the cause for this was that the subject had very long fingernails which were segmented as finger area.Here the plausibility check during the segmentation failed and a capturing of the subject was not possible.To overcome this flaw a fingernail detection could be implemented in the segmentation workflow.

Conclusion
In this work, we proposed a fingerprint recognition workflow for state-of-the-art smartphones.The method is able to automatically capture the four inner-hand fingers of a subject and process them to separated fingerprint images.With this scheme we captured a database of 1,360 fingerprints from 29 subjects.Here we used two different setups: a box setup with constrained environmental influences and a tripod setup.Additionally, we captured touch-based fingerprints as baseline.During a usability study after the capturing the subjects were asked about their experience with the different capturing device types.
Our investigations show that the overall biometric performance of the touchless box setup is comparable to the touch-based baseline whereas the unconstrained touchless tripod setup shows inferior results.All setups benefit from a biometric fusion.A further experiment on the interoperability between touchless and touch-based samples (box setup) shows that the performance drops only slightly.
The presented usability study shows that the majority of users prefer a touchless recognition system over a touchbased one for hygienic reasons.Also, the usability of the touchless capturing device was seen as slightly better.Nevertheless, the user experience of the tested touchless devices can be further improved.
The COVID-19 pandemic also has an influence on the performance and acceptance of fingerprint recognition systems.Here hygienic measures lower the recognition performance and users are more concerned touching surfaces in public areas.
Our proposed method forms a baseline for a mobile automatic touchless fingerprint recognition system and is made publicly available.Researchers are encouraged to integrate their algorithms into our system and contribute to an more accurate, robust and secure touchless fingerprint recognition scheme.

Figure 2 :
Figure 2: Overview of the segmentation of connected components from a continuous stream of input images.

Figure 3 :
Figure 3: Overview of the coarse rotation correction, separation of fingerprint images from each other, fine rotation correction, fingertip cropping and normalization of the fingerprint size.

Figure 4 :
Figure 4: Detailed workflow of the coarse rotation correction.

Figure 5 :
Figure 5: Overview of gray-scale conversion, application of CLAHE and cropping of the fingerprint Region Of Interest (ROI).This process is executed on every separated finger.

Figure 7 :
Figure 7: Distribution of age and skin color according to Fitzpatrick metric [25] of the subjects.

Figure 8 :
Figure 8: NFIQ2.0 score distribution and biometric performance obtained from single finger comparisons.

Figure 12 :
Figure 12: DET curves obtained from the interoperability of different subset of the collected data: Comparison of fingerprints captured with different setups.All captured fingers (finger IDs 2 -5 and 7 -10) are considered.
presents the corresponding DET curves.The touchless box setup shows a good interoperability to the touch-based setup (15.71%).The EER of the open setup again significantly drops (27.27% to touchless box and 32.02% to touch-based).

Figure 13 :
Figure 13: General assessment on fingerprint technology and hygienic concerns.

Figure 14 :
Figure 14: Usability assessment of the touchless and touch-based capturing device in comparison to each other.

2 :
Which type of sensor would you prefer for hygiene reasons?No. 3.1: Which sensor would you prefer regarding its usability?avg.approval in % touch based sensor touchless sensor

Figure 15 :Figure 16 :
Figure 15: Comparative assessment of the capturing device type preference.

Figure 17 :
Figure 17: Illustration of accurate and challenging input images and corresponding result images for sharpness assessment (a, d), segmentation (b, e) and contrast adjustment (c, f).The left images of each block represent an accurate images the right one a challenging one.

Figure 18 :
Figure 18: Illustration of a minutiae-based comparison of two touchless fingerprint samples.The features are extracted using the method described in Section 4. The blue lines indicate mated minutiae.

Table 1 :
Overview on selected recognition workflows with biometric performance.(Device type: P = Prototypical hardware, S = Smartphone, W = Webcam) Figure 1: Overview on the most relevant steps of our proposed method.

Table 2 :
Overview on selected recognition workflows with biometric performance.

Table 3 :
Overview on the NFIQ2.0quality scores and the EER of all captured fingers (finger IDs 2 -5 and 7 -10) separated by sensors.

Table 5 :
Overview on the EER in a fingerprint fusion approach: Fusion over the 4 inner-hand fingers of the left hand (IDs 2 -4) and right hand (IDs 7 -10) fusing and fusion over 8 fingers of both inner hands (IDs: 2 -4, 7 -10).

Table 6 :
Overview on the interoperability of different subset of the collected data: Comparison of fingerprints captured with different setups.All captured fingers (finger IDs 2 -5 and 7 -10) are considered. .