Confidence Intervals for Class Prevalences under Prior Probability Shift

Round 1

Reviewer 1 Report

In this paper, one of the main contributions of this paper is that the  author propose a question “Would it be worthwhile to distinguish confidence and prediction intervals for class prevalences  and deploy different methods for their estimation?”.  The author deeply analyzed those question through simulations. This work is interesting. I only have some comments that I suggest the authors to revise: 1. To better solve those questions, i recommend author to add more large and valuable simulations experiments. 2. The introduction is too short, the author need to add more background. 3. Page 4, line 156, "identical" should be "identically". 4. Page 6, line 192, "The following criteria have..." , The following criteria has. 5. Page 6, line 196, “The methods must be Fisher ”, The methods must be fisher. 6. Some references are not complete. Such as, [6], [12], [20], [22], [23], [27], and [30]. 7. About the latest study of Fuzzy sets, the authors are suggested to cite the following paper. [1] Azadeh, Ali, Morteza Saberi, Nasim Zandi Atashbar, Elizabeth Chang, and Peiman Pazhoheshfar. "Z-AHP: A Z-number extension of fuzzy analytical hierarchy process." In 2013 7th IEEE International Conference on Digital Ecosystems and Technologies (DEST), pp. 141-147. IEEE, 2013. [2] Zhou, Xinyi, Yong Hu, Yong Deng, Felix TS Chan, and Alessio Ishizaka. "A DEMATEL-based completion method for incomplete pairwise comparison matrix in AHP." Annals of Operations Research 271, no. 2 (2018): 1045-1066. [3] Deng, Xinyang, and Yong Deng. "D-AHP method with different credibility of information." Soft Computing 23 [4] Kang, Bingyi, Yong Deng, Kasun Hewage, and Rehan Sadiq. "A method of measuring uncertainty for Z-number." IEEE Transactions on Fuzzy Systems 27, no. 4 (2019): 731-738. Based on these comments, I suggest MINI REVISIONS before its acceptance.

Author Response

Reviewer: In this paper, one of the main contributions of this paper is that the  author propose a question “Would it be worthwhile to distinguish confidence and prediction intervals for class prevalences  and deploy different methods for their estimation?”.  The author deeply analyzed those question through simulations. This work is interesting. I only have some comments that I suggest the authors to revise:

Author: Thank you very much for this kind comment. Please see below for my responses to your comments on details.

Reviewer: 1. To better solve those questions, i recommend author to add more large and valuable simulations experiments. 

Author: The run time for the R-scripts I used for implementing the simulation study is already several hours. I don't want to make replication of the results even harder by again prolonging the run time. That's why instead I have inserted additional wording in the 'Conclusions' section to make it clear that it would be desirable to extend the study and to indicate how this could be done.

Reviewer: 2. The introduction is too short, the author need to add more background. 

Author: I have inserted additional wording in the first paragraph of the introduction to better describe the problem that is investigated in the paper.

Reviewer: 3. Page 4, line 156, "identical" should be "identically". 

Author: I believe "identical" is correct, because the grammar rules here require an adjective, not an adverb.

Reviewer: 4. Page 6, line 192, "The following criteria have..." , The following criteria has. 

Author: "criteria" is the plural of criterion -- see https://dictionary.cambridge.org/dictionary/english/criteria

Reviewer: 5. Page 6, line 196, “The methods must be Fisher ”, The methods must be fisher. 

Author: Fisher is the name of the person who invented this notion of consistency. As a name, its first letter should be a capital letter.

Reviewer: 6. Some references are not complete. Such as, [6], [12], [20], [22], [23], [27], and [30]. 

Author: I have inserted the names of the editors in [6]. [20] is cited in the way suggested by the 'citation()' command in R -- I don't want to modify that. All other references you mention are references to books and, as such, are complete when author, title and year are given.

Reviewer: 7. About the latest study of Fuzzy sets, the authors are suggested to cite the following paper.
[1] Azadeh, Ali, Morteza Saberi, Nasim Zandi Atashbar, Elizabeth Chang, and Peiman Pazhoheshfar. "Z-AHP: A Z-number extension of fuzzy analytical hierarchy process." In 2013 7th IEEE International Conference on Digital Ecosystems and Technologies (DEST), pp. 141-147. IEEE, 2013.
[2] Zhou, Xinyi, Yong Hu, Yong Deng, Felix TS Chan, and Alessio Ishizaka. "A DEMATEL-based completion method for incomplete pairwise comparison matrix in AHP." Annals of Operations Research 271, no. 2 (2018): 1045-1066.
[3] Deng, Xinyang, and Yong Deng. "D-AHP method with different credibility of information." Soft Computing 23
[4] Kang, Bingyi, Yong Deng, Kasun Hewage, and Rehan Sadiq. "A method of measuring uncertainty for Z-number." IEEE Transactions on Fuzzy Systems 27, no. 4 (2019): 731-738.

Author:  Mentioning a potential connection to Fuzzy Sets would blow up the scope of the paper and require a lot more space to give a fair account of the facts. Not even the survey article [5] mentions a connection between prevalence estimation / quantification and Fuzzy Sets. For these reasons, I have not followed your recommendation in this case.

Reviewer: Based on these comments, I suggest MINI REVISIONS before its acceptance.

Author: When amending the paper according to your comments, I have followed the principle to make as few changes as possible.

Reviewer 2 Report

The author did a nice work supported by literature and his own R scripts. I have only detected minor errors reported below. In my view the text is clear and the research well supported.

lines 112-113 I could not find the referred files in the link

https://www.researchgate.net/profile/Dirk_Tasche.

In Appendix A sometimes the signal (minus, sum, and so on) are in the first part of the equation but not when the author changes line: (A18), (A19) and (a21)

Author Response

Reviewer: The author did a nice work supported by literature and his own R scripts. I have only detected minor errors reported below. In my view the text is clear and the research well supported.

Author: Thank you very much for the encouraging comment.

Reviewer: lines 112-113 I could not find the referred files in the link https://www.researchgate.net/profile/Dirk_Tasche.

Author: I have inserted additional wording in the text to make it clearer how to find the R-scripts on my ResearchGate page.
 
Reviewer: In Appendix A sometimes the signal (minus, sum, and so on) are in the first part of the equation but not when the author changes line: (A18), (A19) and (a21)

Author: I have modified the three equations such that when lines are broken the signs appear at the beginning of the new line.