A Comparison of Existing Bootstrap Algorithms for Multi-Stage Sampling Designs
Round 1
Reviewer 1 Report
It is not clear what the motivation is for the paper, and as such it is not clear the contribution of the paper. Is the Rao and Wu (1988) bootstrap method being used? Is this a significant problem?
Author Response
We appreciate your useful and important comments that helped improving our paper. We hope that we have properly understood your comments and that you will find our responses satisfactory. Below, you will find a point-by-point response to your comments/suggestions. Your comments are in normal font and our responses are in italic.
Comments to the Author
It is not clear what the motivation is for the paper, and as such it is not clear the contribution of the paper. Is the Rao and Wu (1988) bootstrap method being used? Is this a significant problem?
Reply: Thank you for this comment. To clarify the objective of the paper, we added the following sentence in the introduction: “The goal of this paper is to compare empirically several existing bootstrap algorithms that have been proposed in the literature for two-stage sampling designs. The bootstrap procedures are compared with respect to bias, stability and coverage probability of confidence intervals.”
Reviewer 2 Report
The paper consist in a thorough presentation and a nice comparison of several bootstrap algorithms for estimating the variance in multi-stage sampling designs.
The algorithms are described in term of bias, stability and coverage.
The paper is concise, but clear and can be published with very minor revisions.
If the Authors have the time to intervene on the paper, the last Section ought to contain a more thorough discussion.
Some points might deserve further deepening, about which we do not ask revision, but only suggest some thought.
Surprisingly, no mention to the software used for the simulation can be found along the paper.
The theoretical contribution of the paper, that can be summarized in formulae (14) and (15), is relegated in the Appendix. Given the structure of the paper, whose Sections 3 and 4 are very short and contain the essentials for arriving at the Section on simulation, the reasons why this occurs can be shared. Formulae (14) and (15) might however be the core of a paper where the content of Sections 3 and 4 is presented in another way.
Section 2.
The reference to equation estimates is very sudden and short; the link with resampling methods is duly mentioned but the reader might feel not helped enough. Some sentences about motivations for estimating the theta’s might be illuminating.
Sections 3 and 4
They are very compact and contain the essential introduction to the simulation study.
At the price of reducing the concision: a) the concept of smooth statistics (parameters?) in Section 3.1 (but at least also 3.4) might deserve some explanation.
Section 6
This short section is not a true discussion. I should like to notice that a contribution of this paper is the comparison of several bootstrap methods that have been proposed along 3 decades. The results of the simulation might therefore be useful for practitioners for the selection of the best method for their datasets. In this regard, a further empirical application might be used to appreciate the differences in results.
The first paragraph contains comments on some methods and not about conclusions about the simulations.
The content of the second paragraph again comments on some methods and not about conclusions about the simulations. Perhaps it ought to constitute a part of an introduction.
Very minor points:
Page 8, Step 4. Check the concordance between “formulae”, that is a plural, and the verb “was” that is singular (also page 9 Step 5, page 10 Step 3, page 13, Step 4, page 15 Step 3, perhaps elsewhere…))
Page 8: is a better citation of the Mirror Match Bootstrap useful?
Page 9, last sentence of Section 3.3 Check the concordance between “a randomization”, that is a singular, and the verb “are” that is plural (also page 11 second line after formula (9))
Page 20. Two contiguous sentences begin with “in practice,…”
Author Response
We appreciate your useful and important comments that helped improving our paper. We hope that we have properly understood your comments and that you will find our responses satisfactory. Below, you will find a point-by-point response to your comments/suggestions. Your comments are in normal font and our responses are in italic.
Comments to the Author
The paper consists in a thorough presentation and a nice comparison of several bootstrap algorithms for estimating the variance in multi-stage sampling designs. The algorithms are described in term of bias, stability and coverage.
The paper is concise, but clear and can be published with very minor revisions.
If the Authors have the time to intervene on the paper, the last Section ought to contain a more thorough discussion.
Some points might deserve further deepening, about which we do not ask revision, but only suggest some thought.
Surprisingly, no mention to the software used for the simulation can be found along the paper.
Reply: We nous mention that the R software was use to carry out the simulation.
The theoretical contribution of the paper, that can be summarized in formulae (14) and (15), is relegated in the Appendix. Given the structure of the paper, whose Sections 3 and 4 are very short and contain the essentials for arriving at the Section on simulation, the reasons why this occurs can be shared. Formulae (14) and (15) might however be the core of a paper where the content of Sections 3 and 4 is presented in another way.
Reply: We have tried to move the some of the material presented in the Appendix in the core of the paper but we felt that the paper was not easy to read. Therefore, we have left Formulae (14) and (15) in the Appendix. Note that Sections 3 and 4 are relatively long.
Section 2.
The reference to equation estimates is very sudden and short; the link with resampling methods is duly mentioned but the reader might feel not helped enough. Some sentences about motivations for estimating the theta’s might be illuminating.
Reply: As requested, we have added some details on parameters defined by estimating equations.
Sections 3 and 4
They are very compact and contain the essential introduction to the simulation study.
At the price of reducing the concision: a) the concept of smooth statistics (parameters?) in Section 3.1 (but at least also 3.4) might deserve some explanation.
Reply: We have clarified the concept of smooth/non-smooth parameter at the end of Section 2.
Section 6
This short section is not a true discussion. I should like to notice that a contribution of this paper is the comparison of several bootstrap methods that have been proposed along 3 decades. The results of the simulation might therefore be useful for practitioners for the selection of the best method for their datasets. In this regard, a further empirical application might be used to appreciate the differences in results.
The first paragraph contains comments on some methods and not about conclusions about the simulations.
The content of the second paragraph again comments on some methods and not about conclusions about the simulations. Perhaps it ought to constitute a part of an introduction.
Reply: We have summarized the findings of the simulation study and added a few additional points.
Very minor points:
Page 8, Step 4. Check the concordance between “formulae”, that is a plural, and the verb “was” that is singular (also page 9 Step 5, page 10 Step 3, page 13, Step 4, page 15 Step 3, perhaps elsewhere…))
Page 8: is a better citation of the Mirror Match Bootstrap useful?
Page 9, last sentence of Section 3.3 Check the concordance between “a randomization”, that is a singular, and the verb “are” that is plural (also page 11 second line after formula (9))
Page 20. Two contiguous sentences begin with “in practice,…”
Reply: Minor comments have been addressed.
Reviewer 3 Report
This is an interesting and well-written paper that compares through simulation experiments several bootstrap variance estimators. I have only a few minor comments below:
1- There is a similar paper on the same topic by Saigo (2010, Journal of Official Statistics) that the authors might want to cite. Could the authors point out if their conclusions are different than those in Saigo (2010)? If so, it would be useful to highlight the new conclusions.
2- Beaumont and Patak (2012) provided a weighted version of the Rao-Wu bootstrap in the same spirit as Rao, Wu and Yue (1992). That weighted version of the Rao-Wu bootstrap would work better for the median than the original version published in JASA. It may not be necessary to add the weighted version of Rao-Wu to the simulation experiment, but the authors might want to mention this point in the discussion of the results.
3- Beaumont and Émond (2022, Stats) developed a bootstrap variance estimation method for any multistage design (not restricted to SRS at each stage or negligible first-stage sampling fractions). The method has just been published so it would be unfair to ask the authors to include it in the simulation study. However, given the authors' paper is intended for the same special issue of the journal, it might be relevant to mention it, perhaps in the discussion section.
4- In the simulation experiments, it is a bit confusing to have f , f_1 and f_2 to represent the first-stage sampling fraction (f_2 is not the sampling fraction at the second stage). The same comment applies to n, n_1 and n_2. Is it really needed to introduce n_1, n_2, f_1 and f_2?
Author Response
We appreciate your useful and important comments that helped improving our paper. We hope that we have properly understood your comments and that you will find our responses satisfactory. Below, you will find a point-by-point response to your comments/suggestions. Your comments are in normal font and our responses are in italic.
Comments to the Author
This is an interesting and well-written paper that compares through simulation experiments several bootstrap variance estimators. I have only a few minor comments below:
- There is a similar paper on the same topic by Saigo (2010, Journal of Official Statistics) that the authors might want to cite. Could the authors point out if their conclusions are different than those in Saigo (2010)? If so, it would be useful to highlight the new conclusions.
Reply: Thank you for pointing out the paper by Saigo. Our conclusions are essentially aligned with those of Saigo (2010). We have added a few sentences in Section 6.
- Beaumont and Patak (2012) provided a weighted version of the Rao-Wu bootstrap in the same spirit as Rao, Wu and Yue (1992). That weighted version of the Rao-Wu bootstrap would work better for the median than the original version published in JASA. It may not be necessary to add the weighted version of Rao-Wu to the simulation experiment, but the authors might want to mention this point in the discussion of the results.
Reply: Thanks for the comment. We have now added Beaumont and Patak (2012) in the section discussing the Rao-Wu method.
- Beaumont and Émond (2022, Stats) developed a bootstrap variance estimation method for any multistage design (not restricted to SRS at each stage or negligible first-stage sampling fractions). The method has just been published so it would be unfair to ask the authors to include it in the simulation study. However, given the authors' paper is intended for the same special issue of the journal, it might be relevant to mention it, perhaps in the discussion section.
Reply: Thanks for the comment. Following your suggestions, we have added a few sentences at the end of Section 6 about the paper of Beaumont and Émond (2022, Stats).
- In the simulation experiments, it is a bit confusing to have f , f_1 and f_2 to represent the first-stage sampling fraction (f_2 is not the sampling fraction at the second stage). The same comment applies to n, n_1 and n_2. Is it really needed to introduce n_1, n_2, f_1 and f_2?
Reply: Thanks for the comment. We have now modified the notation in the simulation study.
Round 2
Reviewer 1 Report
NA
