Biometric Performance as a Function of Gallery Size
, Hal Stern 2, Vladyslav Prokopenko 1, Shagen Djanian 3, Henry Griffith 4 and Oleg Komogortsev 1Round 1
Reviewer 1 Report
This work is interesting and the research design is done well, in showing through synthetic- and real-data experiments that the relationship between the Rank-1 Identification Rate and the gallery size is a log-linear relationship, and also show that a similar relationship does not exist for the Equal Error Rate, which is relatively constant as a function of gallery size.
That said, there is unfortunately one citation that is missing from this paper that seems to be more relevant to this work than any of the other papers cited:
[1] M. Baveja, H. Yuan and L. M. Wein, "Asymptotic Biometric Analysis for Large Gallery Sizes," in IEEE Transactions on Information Forensics and Security, vol. 5, no. 4, pp. 955-964, Dec. 2010, doi: 10.1109/TIFS.2010.2058105.
where the authors of [1] similarly derive the log-linear relationship between the rank-one identification probability and the gallery size, but [1] is not cited in this paper. I think that before this work can be published, it must cite the above paper and address the novelties that their work brings in comparison to [1].
Other than that, I thought this was a good paper and was overall well written. I have the following additional comments / suggestions:
- I think the references following "As a general matter, several authors consider that there is an influence of gallery size on biometric performance" can be removed in favor of discussion of the paper [1] given above.
- It seems that only the threshold is important for increasing gallery sizes. It would be nice to see some discussion on how the Rank-1 IR is threshold-dependent and the ROC curve / EER is threshold-independent.
- The paper states that "For one type of analysis, biometric performance was evaluated for the first 2, 5, 12 or 30 FaceNet PCA components. For another type of analysis, biometric performance was evaluated for the the first 5, 10, 15 or 20 PCA components". Why were different numbers of PCA components selected for the different analyses?
- Why not show the plots with a log scale on the x-axes? It might be easier to see the log-linear relationship this way.
- Why not plot the actual genuine and impostor score distributions, instead of looking at the median and IQR? You could show the general shift of the distributions (and therefore how the thresholds change) for the different gallery sizes.
- It is stated that "The most important predictor of change in EER as additional features are added is in the spread of the impostor similarity score distribution, which becomes narrower as additional features are added", but isn't it both the spread of the genuine and impostor score distributions that affect the EER? It is even stated later that "The ROC-based measures are based on the central tendency and spread of the genuine and similarity score distributions" so maybe this first sentence needs some clarification?
- The format is a bit strange - why is there no conclusions section?
- I didn't see that the acronym "IQR" was spelled out.
- In Algorithm 1, "For each feature j, treat Xijs..." should Xijs be written as a vector Xj? It seems that Xijs is not a function of i and s (since it contains all i's and s's) so not sure why i and s are in the subscript notation.
I think that if these comments can be addressed this paper can be published.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
This paper examined how increases in gallery size affected identification rates and ROC-based measures such as equal error rate (EER). We studied these phenomena with synthetic data as well as real data
from a face recognition study, and report the results. In this paper, the following major points need to be addressed to make paper to an acceptable level.
The main innovation points of this paper are not clear and need to be emphasized in the Introduction.
The titles of figures should be short, and the explanation content of the figure should be placed in the text rather than the figure title. When the figure contains multiple sub figures, each sub figure shall be numbered by using (a), (b), (c), etc. Such as Figure 5, Figure 6, etc.
In Section 5, for the results found in the experiments, it is better to make a summary table to describe them, so that readers can understand them at a glance. The discussion and analysis of the results can be taken as the content at the end of Section 5.
Section 6 It is necessary to provide a conclusion and brief summary of the full text research work, and point out the potential application of the research results of this paper and the future research direction
It is recommended to carefully check the language of the full text. There are some small errors in the text, for example, in line 44: "All error rates rates are expressed...", two "rates".
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
Thank you for addressing the comments. The edits look good to me.
Reviewer 2 Report
The authors have made a great work addressing all my concerns. So, I recommend the paper to be accepted.
