Label-Free Model Evaluation with Out-of-Distribution Detection

Round 1

Reviewer 1 Report

Good paper....It needs some English editing (e.g., 'trian' vs. 'train' in Equation 4, figure 1 rather than Figure 1 in the text, etc.)  You will need higher resolution images as well (and watch the labels..e.g., 'ood' vs. 'OOD.')

Recommendation:  I would recommend providing a diagram of the architecture (easy to do in TensorFlow or Pytorch)....

Solid effort. 

Author Response

Dear Reviewer,

Thank you for your comments and suggestions on our manuscript. We have carefully reviewed your feedback and made the necessary revisions. We appreciate your feedback on the English language issues and have corrected them, including the typos in the figure labels. We have also updated the resolution of the images as per your suggestion.

Regarding the architecture diagram, we apologize for not including the detailed architecture diagrams of Wideresnet and Resnet-44 due to their large size. However, we have provided an overall framework diagram at the beginning of Chapter 3 to illustrate our method.

Here are the revisions we made to the paper：

1.We have made significant adjustments to the English expression in the paper, correcting grammar errors.
2.We have used bitmaps, and the images now do not distort when enlarged. Additionally, we have added more detailed annotations to help understand the content of the images.
3.We have provided a more detailed analysis of other literature and connected it to our work in the introduction and related works.
4.We have provided an overall framework diagram in the method section to help readers better understand our work, and we have revised the statement of the problem definition to avoid ambiguity.
5.We have improved the conclusion section by adding method details, discussing advantages and disadvantages, and providing prospects for future work.
6.We have updated the references section and introduced the latest literature's connection and differences to our work in the paper.

Thank you again for your constructive feedback and for acknowledging our effort. We hope that the revised manuscript meets your expectations .

Best regards

Reviewer 2 Report

Observations regarding Article #2261872

Title: “Label-Free Model Evaluation with Out-of-distribution detection”

General:

-        The article is well organized and structured, and the flow of information is gradually presented to the reader

-        The subject is related to label-free model evaluation that has been developed to estimate the performance of models on un-labelled test sets. The authors of the article propose an Automatic

-        model Evaluation (called AutoEval) method using out-of-distribution data (OOD) detection to reduce the impact of such data on the model evaluation. The authors propose a solution to eliminate OOD using the variance, mean for all image features, and Fréchet distance.

-        In general, the paper is addressed to the advised readers. While in general the style of writing is mostly concise, there are some flaws in phrasing

-        The paper needs intensive check on typing errors, there are some places where there is no space between a sentence and the next one.

Recommendations:

-        To include in the Conclusion section also some possible fields of industry where applicable, or directly present the most recommendable application, in the vision of the authors

-        Also, to include in the Conclusion section some ideas about possible, future developments or related research.

-        To carefully read and correct typing and grammar errors.

Please give more explanation on figure 2, in a comparative mode.

Author Response

Dear Reviewer,

Thank you for your comments and suggestions on our manuscript. We have carefully reviewed your feedback and made the necessary revisions. 

Regarding the issue of typing and grammar errors, we apologize for any mistakes that may have been present in the original manuscript. We have performed an intensive check to eliminate these errors.

In the conclusion section, we briefly summarized the methods proposed in the paper and analyzed the advantages and disadvantages of our proposed method. In response to your suggestions, we have provided descriptions of its applicable scenarios and future prospects. 

We have provided more detailed explanations for all the images in the paper, hoping that this will help to understand the content of the images.

Here are the revisions we made to the paper：

1.We have made significant adjustments to the English expression in the paper, correcting grammar errors.
2.We have used bitmaps, and the images now do not distort when enlarged. Additionally, we have added more detailed annotations to help understand the content of the images.
3.We have provided a more detailed analysis of other literature and connected it to our work in the introduction and related works.
4.We have provided an overall framework diagram in the method section to help readers better understand our work, and we have revised the statement of the problem definition to avoid ambiguity.
5.We have improved the conclusion section by adding method details, discussing advantages and disadvantages, and providing prospects for future work.
6.We have updated the references section and introduced the latest literature's connection and differences to our work in the paper.

Thank you again for your constructive feedback and for acknowledging our effort. We hope that the revised manuscript meets your expectations .

Best regards

Reviewer 3 Report

1. The introduction/literature survey is very short (lines 12 to 29). You need to combine the related work to the introduction

2. Also it is too early to begin explaining the details of your proposed model (starting line 30). This needs to be moved as an introduction to the materials and methods section

3. Line 78, you need a stronger analysis of the other literature and then relate it to your proposed work.

4. Mostly the references in the literature are not very recent (None from 2023, only two from 2022 and the rest are from before.

5. equation on line 85 is incorrect since xi denote the training images. I can see that you wrote a statement that testing is subset of the same set yet this is not the right way to present the split. You need to be more specific as whether this is a 80-20 or 70-30 split and fix the equation.

6. The experimentation seems extensive, yet the 9:1 is usually used with cross validation yet this was not he case. So why didnt you use an ordinary 7:3 split

7- The conclusion needs to have more details of the work presented and the results

Author Response

Dear Reviewer,

Thank you for your valuable comments and suggestions on our manuscript. We have carefully reviewed your feedback and made the necessary revisions. Here is our response to each of your points:

1.We have expanded the content of the Introduction/Literature Survey to provide a more detailed explanation of the problem background, the necessity of solving the problem, and a thorough introduction to the work done by related literature and its limitations.

2.We have revised the manuscript to move the details of our proposed model to the materials and methods section, as you suggested.

3.We have added a stronger analysis of the other literature and related it more closely to our proposed work, as per your suggestion.

4.We have updated the references section with more recent literature to make it more up-to-date, and we have analyzed the similarities and differences between our work and the newly added literature in the paper.

5.We apologize for the confusion caused by our unclear wording. Our intended meaning was that the training set and test set are not identical, and the in-distribution samples in test set that have the same labels as those in the training set, while the out-of-distribution samples in the test set have completely different labels from those in the training set. We have made revisions to the relevant sentences and equations to avoid ambiguity, as suggested.

6.We apologize for the confusion in our previous descriptions. What we meant to convey is that the ratio of in-distribution data (which has the same labels as the training data) and out-of-distribution data (which has labels different from the training data) in the test set is 9:1. We chose this ratio because in real-world scenarios, such OOD data is often only a small fraction of the total data. Additionally, we also conducted experiments with a higher proportion of OOD data, and these experiments are described in Figure 4 of the Results section

7.In the conclusion section, we briefly summarized the methods proposed in the paper and analyzed the advantages and disadvantages of our proposed method. In response to your suggestions, we have described the experimental resultsand have provided descriptions of its applicable scenarios and future prospects. 

Once again, thank you for your valuable feedback. We hope that our revisions have addressed your concerns, and we look forward to your review of the updated manuscript.

Best regards

Round 2

Reviewer 3 Report

Paper is improved but the abstract needs to have more details of the experimentation and results

Author Response

Thank you for your feedback. We appreciate your comment and have updated the abstract to provide more details of our experimentation and results.