Convergence Behavior of Optimal Cut-Off Points Derived from Receiver Operating Characteristics Curve Analysis: A Simulation Study

Round 1

Reviewer 1 Report

Main issue

- Your manuscript would benefit by a validation of the reported results on real data (doi:10.3390/math8101741)

Abstract:

- Briefly present the method used and the main results. 

Introduction

- All information included in this section must be supported by references.

- "several criteria for cut-off point optimality have been proposed" present their advantages and disadvantages.

- "More recently" avoid such expressions (in this particular case is 2012 ... for me it is not recently).

- "included criteria based on sensitivity and specificity" how effective are these criteria; critically evaluate them.

- "more demanding model-based (especially the continual reassessment method) and model-assisted designs (such as Bayesian optimal interval design) are employed" critically evaluate how these models work.

- define "optimal cut-off points". Who say it is optimal. If it is optimal is expected to be only one (no plural).

- Present the state of the art in such way to highlight what is already knowns and where are the gaps.

- "In the following sections, the simulation study is detailed, and the means for investigating the convergence behavior as well as the heuristic and path-based rule are described. The results are followed by a discussion of the findings and concluding remarks close the paper." delete this paragraph because it is not informative and duplicate information.

Materials and Methods

- How were the mean and variances decided? Is there any correspondence in real like for some real measurements? How the type of distributions were decided?

- How the prevalence were decided?

Results

- Fig. 2 - which is the significance of the full stops, whiskers and boxes.

- A table must stay in one page.

- Table 2: Patients, mean It is not clear which variable o you refer. Furthermore, the brackets of 95%CI are round indicating that the value are not included in the range. Is this correct?

- "n=101(50)1,401"?

Discussion

- Present the limitation of your study.

- Discuss the generalizability of the findings.

- Discuss the practical utility of the reported results.

Author Response

Comments and Suggestions for Authors

Main issue

- Your manuscript would benefit by a validation of the reported results on real data (doi:10.3390/math8101741)

AUTHORS’ RESPONSE: A real-life data example has been added (Section 4) as well as the reference (#47), thank you.

Abstract:

- Briefly present the method used and the main results. 

AUTHORS- RESPONSE: done (lines 18-24).

 Introduction

- All information included in this section must be supported by references.

- "several criteria for cut-off point optimality have been proposed" present their advantages and disadvantages.

- "More recently" avoid such expressions (in this particular case is 2012 ... for me it is not recently).

- "included criteria based on sensitivity and specificity" how effective are these criteria; critically evaluate them.

AUTHORS- RESPONSE: The closest-to-(0,1) criterion, the Youden index, and Liu’s index are conceptually different as described in lines 47-54. Their characteristics are further discussed in Section 5.2 (“Optimality of a Cut-off Point”). There is not the best criterion as its choice depends on the researcher’s target; for instance, point closest to perfect discrimination (nearest-to-(0,1)) versus point farthest from no discrimination at all (Youden). An evaluation on top of and beyond the current text touches a different point than our investigation on the convergence behavior for one chosen criterion which, in turn, was chosen for algorithmic stability (Section 2.2). We extended, though, Section 5.2 on the term “optimality”. We deleted “More recently” as requested (line 51).

- "more demanding model-based (especially the continual reassessment method) and model-assisted designs (such as Bayesian optimal interval design) are employed" critically evaluate how these models work.

AUTHORS- RESPONSE: Model-based designs are, to the best of our knowledge, in opposition to up-and-down designs not directly transferable to the issue of cut-off point finding. The Continual Reassessment Method was solely mentioned due to its predominant role in cancer dose-finding studies.

- define "optimal cut-off points". Who say it is optimal. If it is optimal is expected to be only one (no plural).

AUTHORS- RESPONSE: Indeed, we focused on numerical convergence to a known, true value in the given simulation set-ups. The issue of what optimality defines in view of the patients is more challenging and a matter of ongoing debate (see Section 5.2). Moreover, the choice of any cut-off point does also depend on how the ROC curve is generated. To this end, we have extended the discussion by a section on limitations of our study (Section 5.4).

- Present the state of the art in such way to highlight what is already knowns and where are the gaps.

AUTHORS- RESPONSE: The original idea here is to transfer a design known from cancer dose-finding trials (up-and-down designs like the Traditional Escalation Rule, also known as “3+3”) to the issue of cut-off point finding and to investigate the convergence behavior of the chosen cut-off point with increasing sample size under different scenarios.

- "In the following sections, the simulation study is detailed, and the means for investigating the convergence behavior as well as the heuristic and path-based rule are described. The results are followed by a discussion of the findings and concluding remarks close the paper." delete this paragraph because it is not informative and duplicate information.

AUTHORS- RESPONSE: done (lines 71-72).

Materials and Methods

- How were the mean and variances decided? Is there any correspondence in real like for some real measurements? How the type of distributions were decided?

- How the prevalence were decided?

AUTHORS- RESPONSE: Distributional forms and prevalence were inspired by similar investigations and are supposed to cover broadly. We had no specific marker in mind which, for the sake of this investigation, neither is necessary. Varying the prevalence across 0.1, 0.3, 0.5, and 0.7 suffices to investigate both the situation of cases and controls contributing alike (0.5) and notable deviations from 0.5. Clinically, a smaller prevalence is more likely than a larger one; therefore, a prevalence of 0.9 is not of interest.

Results

- Fig. 2 - which is the significance of the full stops, whiskers and boxes.

AUTHORS- RESPONSE: The boxes show 1^st and 3^rd quartile as they usually do in applied sciences. The whiskers indicate the smallest and largest value within the fences that are defined as 1^st quartile minus 1.5 times the interquartile range and 3^rd quartile plus 1.5 times the interquartile range. The dots are measurements that fall outside these fences. As the definition of the whiskers varies in the applied sciences, this definition was already mentioned in the first submission (Section 2.4). However, we have extended the legend of Figure 2 to this end to support the figure’s clarity.

- A table must stay in one page.

AUTHORS- RESPONSE: Acknowledged, thanks.

- Table 2: Patients, mean It is not clear which variable do you refer. Furthermore, the brackets of 95%CI are round indicating that the value are not included in the range. Is this correct?

AUTHORS- RESPONSE: The title indicates that the mean number of patients (95% CI) of cut-off points derived by the heuristic algorithm I are shown in Table 3. “Patient, mean (95% CI)” has been changed to “Mean no. of patients, 95% CI” in Table 3 and Table 4. Moreover, square brackets are used now to indicate that the confidence limits themselves are actually included. Thank you.

- "n=101(50)1,401"?

AUTHORS- RESPONSE: This has been changed to n=101,151,201,…,1,401 in the revised manuscript.

Discussion

- Present the limitation of your study.

AUTHORS- RESPONSE: Please see the new Section 5.4.

- Discuss the generalizability of the findings.

- Discuss the practical utility of the reported results.

AUTHORS- RESPONSE: Please see Section 5.3 and, especially, the new Section 5.4 on the limitations of our study.

Reviewer 2 Report

My comments are attached here.

Comments for author File: Comments.pdf

Author Response

Review of “Convergence behavior of optimal cut-off points derived from receiver operating characteristics curve analysis: a simulation study”

Date: Oct 22, 2022

The authors aimed to study the convergence of the optimal cutoff point and explored fixed and heuristic path-based algorithms in determining the ideal sample size. The authors have investigated the effect of sample size on the variability of the cutoff point by choosing the fixed total sample size of the healthy and diseased populations. Additionally, the authors have also explored 2 heuristic algorithms where the total sample size is not fixed and chosen based on when the optimal cutoff estimate is within the 1% of the true estimate. These directions are important and could be important in the future study designs. However, I think that the manuscript could be improved substantially.

AUTHORS’ RESPONSE: Thank you very much for your positive and constructive feedback. Much appreciated.

I have listed all my comments (both major and minor) below:

Major comments

• Apparently, the authors have implemented Liu’s methodology in assessing the convergence of the cutoff point. I wish they had briefly described the methodology in the paper for the better understanding of the readers.

AUTHORS’ RESPONSE: Section 2.3 on the evaluation of the true optimal cut-off points for the closest-to-(0,1) criterion, Liu’s method, and the Youden index in all scenarios has been extended to also show the definition of these three criteria. The discussion on “optimality” (Section 5.2) has been extended. We would like to stress that we implemented the closest-to-(0,1) criterion and actually only that one (see Abstract as well as Suppl. Materials 1, macro ‘Evaluation’, line 340: cutpt status marker if …, nearest). This point has been stressed in the new limitations of the study section (5.4), see lines 360-362.

• In introduction, the authors mentioned Agatston score which could range from 0 – 400 or higher, yet none of the chosen simulation setup can cover that much of range.

AUTHORS’ RESPONSE: True; however, taking the diversity of biomarker distributions into consideration, we aimed for four diverse settings, inspired by previous, similar set-ups (Section 2.1). As per request of you and another reviewer, we have, though, added a real-life data example using the Agatston score (Section 4). Figure A2 underlines the extreme skewness to the right in both D0 (healthy population) and D1 (diseased population).

• In simulation, the authors have considered 4 different scenarios with varying distributional assumptions for subjects with and without target condition and used the Liu’s method to compare the performance in those scenarios. The estimation of cutoff points can significantly vary with the shape of the ROC curve. Especially when the ROC curve is estimated empirically (for smaller sample size or for cases with extreme biomarker distributions) the cutoff point could be different as compared to when the ROC curve is estimated as a smooth curve (often parametrically and semi-parametrically). The shape of the ROC curve (concave or non-concave) can also impact the cutoff estimation. In short, the estimation process of the ROC curve will affect the cutoff estimate and thus the convergence pattern could also vary with respect to the ROC estimation. I think this is an important direction the authors should incorporate to make the findings more significant.

AUTHORS’ RESPONSE: Point taken, thank you. We have extended Section 5.2 on the term “optimality” and added the limitations of our study, incorporating your point (Section 5.4, lines 352-360).

• Section 2.2. It is confusing which way the authors estimated the cutoff point. In Section 2.2, the authors mentioned that the closest-to-(0,1) criterion was used. In Section 4.2 (lines 235 – 238), the authors cited other authors who advised against the closest-to-(0,1) criterion and suggested Youden’s index. Later in the next page, the authors justified (lines 256 – 259) that using the closest-to-(0,1) criterion is sufficient to investigate the convergence pattern. However, the authors have shown that (Table 1) the difference between Liu’s method and closest-to-(0,1) criterion could be more than 10%. However, the Liu’s method and Youden index have comparable performances.

AUTHORS’ RESPONSE: We agree, but we needed to choose the closest-to-(0,1) criterion out of algorithmic stability (Section 2.2) and to be able to generate our results. The discussion on “optimality” in Section 5.2 does, indeed, accentuate that both Liu’s method and the Youden index offer – in opposition to the closest-to-(0,1) criterion – a clinically meaningful interpretation. We have elaborated on the term “optimality” also in a wider sense as other researchers have aimed at trichotomizations, introducing an interval of uncertainty or a grey zone of transition regarding the overlap of the distributions of D0 and D1 (Section 5.2, lines 318-325). We have also added the limitation of our study that the work is based on one specific criterion (namely, closest-to-(0,1)) and one specific nonparametric ROC curve estimation, see lines 360-362.

• Section 2.5. While illustrating the heuristic and path-based algorithms, the authors have proposed incrementing the sample sizes by 50 and 100. The authors don’t justify the choice of 50 and 100. Additionally, the authors should have described the process in more detail. Do the authors mean that the process will stop at the sample size when the corresponding cutoff estimate is within the 1% deviation from the true cutoff?

AUTHORS’ RESPONSE: The increments of 50 and 100 were chosen out of practicability considerations as the total sample size should preferably be in the range of 400 to 600 observations. With varying prevalence, especially with a prevalence as low as 0.1, smaller increments than 50 may imply that no cases are added from one iteration to the next.

The process is described in Section 2.5, especially lines 146-151. However, we have rephrased the beginning of Section 3.2 (lines 196-198) in which the first results of the heuristic algorithm are described. We hope that this improves the clarity. The algorithm stops as soon as the cut-off point evaluated with n subjects deviates less than 1% from the precedent estimate which was evaluated with n-50 (or, for heuristic algorithm II, n-100) subjects before. The true cut-off point cannot be used as stopping criterion as it is unknown (to the algorithm).

• Figure 3 is slightly confusing. Shouldn’t the cutoff corresponding to the vertical lines be within 1% deviation limits when the vertical line (dotted or dasheddotted) intersects the horizontal lines? In other words, it seems the vertical lines should correspond to the sample size where the cutoff point is within the 1% deviation for the first time. However, in most of the cases (for example, top left panel) the cutoff estimate comes within the 1% deviation limits for the first time for sample size = 701, yet the optimal sizes are chosen as 151 and 201. The authors should illustrate the figure more clearly.

AUTHORS’ RESPONSE: We acknowledge that Figure 3 is challenging, but wanted to illustrate both the target area where the chosen cut-off point hopefully falls into (namely 1% deviation from the true value; horizontal lines) and the sample sizes at which the heuristic algorithms actually stop (vertical lines). We emphasized that even with N=1,401 only 5 out of 9 trials end up with a chosen cut-off point that actually deviates less than 1% from the true (but to the algorithm unknown) optimal cut-off point (lines 226-230).

• Figure A1. Why did the authors use 10000000 observations?

AUTHORS’ RESPONSE: We employed that many observations to accurately estimate the empirical ROC curves for scenarios 1-4 (Figure A1). Earlier, we made use of the byproduct that the true optimal cut-off points can be estimated accurately as well, but learned that the true cut-off points can be easier and more elegantly derived by making use of the true, underlying distributions of D0 and D1 in Scenarios 1-4. The Stata source codes have been extended to this end (lines 169-207 in Suppl. Mat. S1).

Minor comments

• Figure 2 is more informative. The authors could provide the same for the other scenarios and could put them in supplement/appendix.

AUTHORS’ RESPONSE: Done. See Supplementary Materials S3.

• Maybe change “Patients” to “Number of patients” in the heading for all Tables 2-4.

AUTHORS’ RESPONSE: Done, thank you.

• A real data application would have been nice.

AUTHORS’ RESPONSE: Done, see Section 4.

• Shouldn’t the literature review in Section 4.2. be in the introduction section?

AUTHORS’ RESPONSE: Indeed, it would be an alternative to extend the introduction this way. However, we prefer the introduction to be brief, leaving the more extensive elaborations for the discussion. After all, we even expanded Section 5.2 on the term “optimality” and think that the current structure is easier for the reader.

Reviewer 3 Report

The authors study, via Monte Carlo simulations, the behavior of the Youden estimator when the sample sizes increase. They play with different parameters and different prevalence.

The paper is well-written but I think the authors should dig in the theoretical results beyond to continue with their research. I mean, they say “Optimal cut-off points are often derived from fixed sample size studies”. For sure, in practice, you have a fixed sample size and you work with it. Theoretically, asymptotic behavior of Youden index is well known (see, for instance, NONPARAMETRIC METHODS FOR EVALUATING DIAGNOSTIC TESTS on JSTOR). Conventionally, the study of the finite-sample behavior of any estimator, in general, based on Monte Carlo simulations, includes different sample sizes. If the target is to present results useful for sample sizes consideration in cohort studies, they should consider variability of the samples, prevalence (they already done), and shape (they already done), and to provide an adequate table with these results. If you want a precision of XX, your prevalence is going to be [..], the variance and the shape of your distributions are […] then you should have XX sample.

Pag 2. Lin 46-47. The concept of optimality is always complex. Obviously, the two criteria the authors highlight, are optimal in their own sense but always open to discussion. Any proposed criterion is optimum from its own approach. This issue is again in the beginning of subsection 2.3 in which the authors talk again about “true optimal cut-off points” as if this would be an objective condition. You are checking for optimal cut-off points under a particular criterion.

Pag 2. Lin 56. Beyond the paper introducing the R package. Lopez-Raton has several papers in which she discussed the methodological aspects of some of the procedures. Her PhD dissertation was devoted to this topic. Perhaps all these references deserve some comments in this paper.

Pag 2. Lin 56-57. I find the jump between the two paragraphs is too much. Perhaps the authors should improve the connection between them.

Pag 3. Lin 116. Why you are computing the real points in this way? You have real expressions for the distributions so, you can compute the optimal thresholds using the true equations.

Author Response

Comments and Suggestions for Authors

The authors study, via Monte Carlo simulations, the behavior of the Youden estimator when the sample sizes increase. They play with different parameters and different prevalence.

The paper is well-written but I think the authors should dig in the theoretical results beyond to continue with their research. I mean, they say “Optimal cut-off points are often derived from fixed sample size studies”. For sure, in practice, you have a fixed sample size and you work with it. Theoretically, asymptotic behavior of Youden index is well known (see, for instance, NONPARAMETRIC METHODS FOR EVALUATING DIAGNOSTIC TESTS on JSTOR). Conventionally, the study of the finite-sample behavior of any estimator, in general, based on Monte Carlo simulations, includes different sample sizes. If the target is to present results useful for sample sizes consideration in cohort studies, they should consider variability of the samples, prevalence (they already done), and shape (they already done), and to provide an adequate table with these results. If you want a precision of XX, your prevalence is going to be [..], the variance and the shape of your distributions are […] then you should have XX sample.

AUTHORS’ RESPONSE: Thank you very much. We have added references on the asymptotic behavior of the Youden index (#51,52), but we need to underline that we employed the closest-to-(0,1) criterion for optimality (see Abstract as well as Suppl. Materials 1, macro ‘Evaluation’, line 340: cutpt status marker if …, nearest). To this end, we extended Section 5.2. The purpose of the study was to examine whether our heuristic algorithm would be capable of proposing an optimal cut-off point with sufficient accuracy. Admittedly, we have been quite conservative in aiming for an optimal cut-off point that deviates less than 1% from the true (and to the algorithm unknown) value.

Pag 2. Lin 46-47. The concept of optimality is always complex. Obviously, the two criteria the authors highlight, are optimal in their own sense but always open to discussion. Any proposed criterion is optimum from its own approach. This issue is again in the beginning of subsection 2.3 in which the authors talk again about “true optimal cut-off points” as if this would be an objective condition. You are checking for optimal cut-off points under a particular criterion.

AUTHORS’ RESPONSE: Thank you, much appreciated! We extended the respective Section 5.2 on the term “optimality” (and what that constitutes) and added Section 5.4 on the limitations of our study. The latter emphasizes that our work is based on one specific criterion (closest-to-(0,1)) and one specific nonparametric estimation of the ROC curve, namely the empirical one as implemented in Stata.

Pag 2. Lin 56. Beyond the paper introducing the R package. Lopez-Raton has several papers in which she discussed the methodological aspects of some of the procedures. Her PhD dissertation was devoted to this topic. Perhaps all these references deserve some comments in this paper.

AUTHORS’ RESPONSE: Thank you. We acknowledge the work of Lopez-Raton and colleagues on the symmetry point in Section 5.2 and have added references #29,30.

Pag 2. Lin 56-57. I find the jump between the two paragraphs is too much. Perhaps the authors should improve the connection between them.

AUTHORS’ RESPONSE: The jump is, unfortunately, inevitable; however, we rephrased line 60 (“In cancer research, …”), hoping that the analogy to the beginning of the previous paragraph in line 47 (“In diagnostic research,…”) improves the readability. The paper is, after all, bridging an idea from dose-finding in early phase cancer trials with the classical challenge of cut-off point determination in diagnostic research.

Pag 3. Lin 116. Why you are computing the real points in this way? You have real expressions for the distributions so, you can compute the optimal thresholds using the true equations.

AUTHORS’ RESPONSE: Good point! The optimal thresholds were a byproduct of the empirical ROC curves shown in Figure A1. However, we have now assessed these using the available distributions for D0 and D1 in all scenarios. Section 2.3 has been revised to this end, and respective Stata codes have been added to the Supplementary Materials S1 (lines 169-207). This is, indeed, much more efficient and elegant. Thank you.

Round 2

Reviewer 1 Report

Your manuscript looks better.

Please solve the following issues before publication:

- Move the text from the Results in lines 242-249 and 263-264... to the Methods section, by creating a new suvsectiob 2.6.

- Provide the 95%CI for AUC=0.73.

- In Reference section, the number 46 is listed but the details of references are missing.

Author Response

Your manuscript looks better.

Please solve the following issues before publication:

- Move the text from the Results in lines 242-249 and 263-264... to the Methods section, by creating a new subsection 2.6.

AUTHORS’ RESPONSE: Done. Thank you.

- Provide the 95%CI for AUC=0.73.

AUTHORS’ RESPONSE: Done, see lines 256-7.

- In Reference section, the number 46 is listed but the details of references are missing.

AUTHORS’ REPONSE: Rechecked and corrected. Thank you. 

Reviewer 2 Report

My comments are attached herewith.

Comments for author File: Comments.pdf

Author Response

Review of “Convergence behavior of optimal cut-off points derived from receiver operating characteristics curve analysis: a simulation study”

Date: November 2, 2022

Thanks to the authors for carefully addressing all my concerns. The manuscript has shaped up nicely. I still have some concerns, which I think can be easily handled by adding/rephrasing some discussions. I have pointed them below:

• Section 4. Thanks for adding the real-life example. I am guessing that the cut-off point 184.7 was calculated using fixed sample size. I don’t think calling it a “true” cut-off point is appropriate given that it is a real data.

AUTHORS’ RESPONSE: Thank you. We renamed the “true” optimal cut-off point the empirical optimal cut-off point based on the full dataset instead.

• It seems from the Figure 1 that higher score means with target condition which implies that the higher score than cut-off will be classified as “with target”. It is helpful to add something like that in the Section 4 so that the readers understand that any future score > 184.7 means potential cardiovascular disease.

AUTHORS’ RESPONSE: Thank you. We added that larger values of the Agatston score are associated with increased risk (lines 254-5).

• Figure 3. I still have concerns with it. I think what is missing from the caption/description of the plot is the black solid line. The authors have written the caption as “Line plots of chosen cut-off points for the first nine trials…”. My guess is that the authors calculated the cut-off points at all sample size n = 101, 151, …,1401 using fixed sample size algorithm and showed (by 2 different vertical lines) where the cut-off estimates are within 1% of the corresponding precedent estimate. Hypothetically, those will be the sample sizes where the heuristic algorithms will stop. I think adding few more lines will clearly illustrate the heuristic algorithm and the interpretation of Figure 3.

AUTHORS’ RESPONSE: An explanation for the dark blue lines has been added, and the caption has been extended as proposed (lines 218-23). Thank you.

• Also, the prevalence level is missing from the caption of Figure 3.

AUTHORS’ RESPONSE: The prevalence of 0.5 has been added to the caption (lines 218-9).

• Figure 4. The authors should illustrate how they implemented the heuristic algorithm for the real data. Did they draw random sample of size = n (101, 151, …, 17251) out of the total sample (17,251) with a given prevalence of 15%? The steps are missing.

AUTHORS’ RESPONSE: We have extended the description of the real-life example accordingly (lines 161-2, lines 259-61).

• One minor suggestion: Have the authors thought about a different set of heuristic algorithms which will stop only when the cut-of is within the target area for the first time. Those algorithms might have less bias for cases with extreme prevalence (i.e., 0.1 or 0.7) as compared to the algorithms proposed by the authors, however, the sample size will increase on an average. Running a simulation based on these won’t be a problem. However, that approach will be based on a set “true optimal cut-off” which is unknown for a real data. But the authors can take a similar route of using a fixed sample size-based cut-off as the reference target area. I’m raising this issue since I feel these heuristic approaches have a real potential in designing clinical trials, however, performs slightly poorly as compared to the fixed sample size for cases with extreme prevalence. The authors can think about it and may discuss the consequences if they find reasonable.

AUTHORS’ RESPONSE: Indeed, we experimented, for instance, with an extended heuristic rule demanding three consecutive proposed cut-off points to deviate less than 1% from each other, but did not observe satisfactory improvement considering the increasing ‘costs’ in terms of increased sample sizes. A closer look at Figure 3 in this regard visualizes actually as much. We thought of the split half method, but did neither see an improvement as ‘optimal’ cut-off points are then based on smaller sample sizes and averaging these seems counterproductive. It is true that there are many possibilities for simple heuristic rules and likewise more sophisticated approaches, maybe a Bayesian one that builds on a clinically meaningful prior distribution for the optimal cut-off point whereas ROC-derived values investigate the whole measurement range irrespective of clinically relevant and meaningful areas. In general, an extreme prevalence of 0.1 or 0.7 is a challenge to any cut-off point finding procedure. However, there is plenty of room for future endeavors. Thank you for your encouraging comment, much appreciated!

Reviewer 3 Report

I think the authors have done a really good jobs responding to my previous questions/concerns. I think they have shown a deep knowledge of the topic they a dealing with. The paper is clearer, and I think it is good enough for publication.

Author Response

I think the authors have done a really good jobs responding to my previous questions/concerns. I think they have shown a deep knowledge of the topic they a dealing with. The paper is clearer, and I think it is good enough for publication.

AUTHORS’ RESPONSE: Thank you very much for your efforts and your positive feedback. Much appreciated.