Improving Re-Identification by Estimating and Utilizing Diverse Uncertainty Types for Embeddings

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The idea is interesting yet methods are not clear. The study design should be cleared. The sections include unnecessary details, which causes loss of interest and makes it hard to follow the study. The mathematical details can be described further. The sections also include some repetition.

1. Please comprehensively revise the abstract to reflect the study and findings clearly.

2. Section 2 needs to be revised and shortened. The section reads like a proposal. You're providing details which causes lost of attention.

3. Please remove the paragraph starting at 244. It's unnecessary and removing it not changing the structure.

4. Please revise sections 3.1 and 3.2, which are unnecessarily long. You're repeating some information from the previous section.

Author Response

Thank you very much for taking the time to review this manuscript and for your detailed comments, which helped to improve the manuscript. Please find below the detailed responses.

Comment 1: Please comprehensively revise the abstract to reflect the study and findings clearly.
Response 1: Thank you for pointing this out. We have revised the abstract and added details about our findings.

Comment 2: Section 2 needs to be revised and shortened. The section reads like a proposal. You're providing details which causes lost of attention.
Response 2: We partially agree. We have removed statements that are already included in the introduction or further explained in Section 3. Other than that, we believe that a comprehensive review of the state of the art should be part of a journal publication. The headings make it easy for readers to easily skip paragraphs they are already familiar with or to only skim this section.

Comment 3: Please remove the paragraph starting at 244. It's unnecessary and removing it not changing the structure.
Response 3: We disagree. Section 3 is a long section. Therefore, we use this section introduction to provide an overview of where the reader can find what information. If one wants to read the manuscript from top to bottom, this short paragraph can easily be skipped.

Comment 4: Please revise sections 3.1 and 3.2, which are unnecessarily long. You're repeating some information from the previous section.
Response 4: We have revised these subsections and found no repetition of information other than in introductory sentences. Furthermore, shortening these subsections would reduce the quality of the manuscript. We would have to include only the formal part and remove the examples that provide a good intuition of the topic. We understand that Section 3.2 may seem a bit too long for an informed reader. However, for a reader new to the topic of uncertainty types, it provides the necessary knowledge and intuition through examples. Therefore, we had already included the statement "Readers who are already familiar with this topic may choose to skip this subsection" at the beginning of Section 3.2.

The document highlighting the differences to the previous version of the manuscript is attached.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

As my ratings above indicate, I am very happy with the quality of this submission - I enjoyed reading it. The most important thing, the content or rather the technical contribution of the work, is relevant, sufficiently novel, and certainly of interest to the journal's readership. It is also clearly described, both when it comes to the verbal aspect of the exposition, as well as the rigorous formalization of the proposed method.

The quality of writing is also very good as are the organization and the flow of the manuscript. The figures included are also professional, clear, and attractive, and they meaningfully aid the reading process. The results are convincing and comprehensive.

I only have minor comments to give, though some of them really must be made; others should be taken as suggestions. 

- line 4: "a first" -> "the first"

- lines 4-5: I suggest that the sentence fragment "namely model uncertainty also known as epistemic uncertainty, data uncertainty, also known as aleatoric uncertainty" is re-written as follows, for clarity: "namely model uncertainty (also known as epistemic uncertainty), data uncertainty (also known as aleatoric uncertainty)"

- line 59: "as in [25]" is not how citation should be done. Citations are not a part of the text but should be rather considered as meta-information. "as in [25]" reads exactly the same as "as in" which makes the sentence nonsensical; rather, the authors should write "as by Dou et al. [25]"; the same mistake can be found throughout the manuscript and all of these should be corrected

- by line 144 it becomes clear that the authors are using far too much emphasis; when a significant portion of text on a page is italicized, one's eyes strain, reading becomes difficult, and the attempt to emphasise so much results in nothing being paid different attention to, i.e. there is no emphasis; if you desire, emphasise a term when it is first introduced and, I would recommend, only if truly needed; thereafter no emphasis is needed or is useful

- line 185: in "Utilization of Uncertainty during Inference" and possibly elsewhere (please check), the authors should ensure that they are adopting either sentence of title case; if the latter, the title should read "Utilization of Uncertainty During Inference"

- figure 1: something is wrong here as (a) and (b) look the same; please check; also note the citation style issue I highlighted previously in the caption

- line 297 (and elsewhere): the varied text background colours should be removed

Author Response

Thank you very much for taking the time to review this manuscript and for providing such a positive feedback and valuable comments to improve the manuscript. The detailed responses are provided below.

Comment 1: line 4: "a first" -> "the first"
Response 1: Thank you for pointing this out. We have incorporated this change.

Comment 2: lines 4-5: I suggest that the sentence fragment "namely model uncertainty also known as epistemic uncertainty, data uncertainty, also known as aleatoric uncertainty" is re-written as follows, for clarity: "namely model uncertainty (also known as epistemic uncertainty), data uncertainty (also known as aleatoric uncertainty)"
Response 2: Thank you for pointing this out. We have also adopted this proposed amendment.

Comment 3: "as in [25]" is not how citation should be done. Citations are not a part of the text but should be rather considered as meta-information. "as in [25]" reads exactly the same as "as in" which makes the sentence nonsensical; rather, the authors should write "as by Dou et al. [25]"; the same mistake can be found throughout the manuscript and all of these should be corrected
Response 3: Thank you for bringing this to our attention. Where appropriate, we have added the authors' names or the cited method to address this issue.

Comment 4: by line 144 it becomes clear that the authors are using far too much emphasis; when a significant portion of text on a page is italicized, one's eyes strain, reading becomes difficult, and the attempt to emphasise so much results in nothing being paid different attention to, i.e. there is no emphasis; if you desire, emphasise a term when it is first introduced and, I would recommend, only if truly needed; thereafter no emphasis is needed or is useful
Response 4: We agree. We have removed most of the emphasis and reduced it to an appropriate amount.

Comment 5: line 185: in "Utilization of Uncertainty during Inference" and possibly elsewhere (please check), the authors should ensure that they are adopting either sentence of title case; if the latter, the title should read "Utilization of Uncertainty During Inference"
Response 5: Thank you for pointing this out. We have corrected this error throughout the manuscript.

Comment 6: figure 1: something is wrong here as (a) and (b) look the same; please check; also note the citation style issue I highlighted previously in the caption
Response 6: Thank you for bringing this to our attention. This was an error on the build server caused by the naming of the included images. Renaming the images fixed the problem.

Comment 7: line 297 (and elsewhere): the varied text background colours should be removed
Response 7: We partially agree. We have reduced the use of text background colors to a minimum. Now, we only use text background colors for the specific text where the color scheme is introduced for the mathematics in Section 3.1 and for the architecture in Figure 3. In our figures, however, we believe that the color serves as valuable information that makes it much easier to understand what is being displayed. In our first draft of the figures, we did not include color highlighting, and it was very difficult to distinguish the information provided by the various sub-figures, especially at a glance. Therefore, we decided in favor of highlighting different uncertainty types and the feature vector with consistent colors.

The document highlighting the differences to the previous version of the manuscript is attached.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The paper titled "Improving Re-Identification by Estimating and Utilizing Diverse Uncertainty Types for Embeddings" by Markus Eisenbach et al. explores how to improve the performance of feature embedding vectors in re-identification (Re-ID) tasks by estimating and utilizing model uncertainty, data uncertainty, and distributional uncertainty. The researchers propose an architecture based on Uncertainty-Aware Learning (UAL) that estimates these three types of uncertainty through Bayesian modules and embedding heads, and validates the approach through experiments on the Market-1501 dataset. The experimental results show that utilizing uncertainty information can significantly enhance re-identification performance, especially by adjusting feature vectors with model uncertainty, which can further improve performance. In addition, distributional uncertainty can effectively estimate whether an input sample is outside the distribution of training data.

 

The shortcomings include:

1. Generalization ability: The study is mainly based on the Market-1501 dataset, and it may need to be verified on more diverse datasets to test the generalization ability of the method.

2. Uncertainty interpretation: Although the paper proposes methods for estimating uncertainty, the specific impact of these uncertainties on model decision-making and feature vectors may not be fully explained.

3. Practical application: Although the paper proposes a new method, it does not elaborate on its deployment and application in practical autonomous systems.

4. Comparison benchmark: The comparison with existing technologies in the paper is mainly focused on re-identification tasks, and it may need to be compared with a wider range of uncertainty estimation methods to demonstrate its advantages and limitations.

5. User interaction: How to convey and explain uncertainty information to users in applications involving human interaction, and how this affects users' trust and acceptance of the system, is an area not covered by the paper.

 

References:

Shi L, Zhang R, Wu Y, et al. AHC-Net: a road crack segmentation network based on dual attention mechanism and multi-feature fusion[J]. Signal, Image and Video Processing, 2024: 1-12.

 

Comments on the Quality of English Language

Good

Author Response

Thank you very much for taking the time to review this manuscript and for pointing out some potential shortcomings. Below are the detailed responses.

Comment 1: Shortcoming generalization ability: The study is mainly based on the Market-1501 dataset, and it may need to be verified on more diverse datasets to test the generalization ability of the method.
Response 1: We agree. Therefore, we have already discussed this in the future work section.

Comment 2: Shortcoming uncertainty interpretation: Although the paper proposes methods for estimating uncertainty, the specific impact of these uncertainties on model decision-making and feature vectors may not be fully explained.
Response 2: We disagree. The utilization of diverse uncertainty types for better decision-making by improving the feature vector or by flagging inputs as being out-of-distribution is the main focus of this paper and is extensively explored in Section 4.

Comment 3: Shortcoming practical application: Although the paper proposes a new method, it does not elaborate on its deployment and application in practical autonomous systems.
Response 3: Thank you for pointing this out. This paper has a methodological and algorithmic focus that fits the journal. Exploring the practicality on a real system is beyond the scope of this paper and of the topics associated with this journal. We leave that to future research. Therefore, we have added a statement in the future work section.

Comment 4: Shortcoming comparison benchmark: The comparison with existing technologies in the paper is mainly focused on re-identification tasks, and it may need to be compared with a wider range of uncertainty estimation methods to demonstrate its advantages and limitations.
Response 4: We disagree. The title of the paper clearly indicates the focus on re-identification. The transfer of our method to other applications is left to future work. Therefore, we already had a statement in the future work section. Regarding other methods: We have included all state-of-the-art methods for which the authors provide the source code. This includes methods from face recognition.

Comment 5: Shortcoming user interaction: How to convey and explain uncertainty information to users in applications involving human interaction, and how this affects users' trust and acceptance of the system, is an area not covered by the paper.
Response 5: Thank you for bringing this to our attention. We believe that this should be evaluated in future work, as including such an analysis would go beyond the focus of this work. Therefore, we have added a statement in the future work section.

In summary, we appreciate that you have pointed out several directions for future research. We have added the missing points in the future work section.

The document highlighting the differences to the previous version of the manuscript is attached.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

I am happy with the authors' revisions and responses. Where disagreements (minute!) remain, considering their nature (mostly aesthetic), I consider it the authors' prerogative to have the final word.

Reviewer 3 Report

Comments and Suggestions for Authors

Good