Inferential Statistics Is an Unfit Tool for Interpreting Data

Round 1

Reviewer 1 Report

Title of Article: Inferential statistics is an unfit tool for interpreting data

Journal: Applied Sciences

MS#: applsci-1794084

 

Although the article says little that has not been said before, I am in favor of moving in the direction of eventual publication anyhow. The reason is that the main lesson is extremely important and bears repeating in different ways, as most researchers still seem not to have learned it. Thus, the manuscript potentially contributes.

 

That said, there are places where the manuscript falls short, but these can be improved. Thus, what I say below ought to be within the capability of the author to handle in a revision and are not a reason for permanent rejection.

 

1.     The author rightly points out the many assumptions but needs more specificity about exactly what they are or at least the categories of assumptions. Absent that specificity, the naïve researcher whom the author is attempting to reach will be unlikely to be convinced about just how weak whole inferential models are. I have proposed a taxonomy of such assumptions and, at the risk of being self-serving, it could really help the author with this. The reference is below.

 

Trafimow, D. (2019). A taxonomy of model assumptions on which P is based

and implications for added benefit in the sciences. International Journal of Social Research Methodology, 22(6), 571-583. doi: 10.1080/13645579.2019.1610592

 

2.     The author seems to like confidence intervals a lot, but these are quite problematic too. First, a 95% confidence interval does NOT indicate there is a 95% probability that the parameter is in the interval. And there are other problems associated with that (see reference below). Second, the inferential model is just as wrong for confidence intervals as for P-values. Third, if one is interested in estimation, the a priori procedure is much superior to traditional confidence intervals. The article below explains all this.  I’d like to see a better treatment that is clear about the negatives associated with confidence intervals.

 

Trafimow, D. (2019). A frequentist alternative to significance testing, p-values, and

confidence intervals. Econometrics, 7(2), 1-14. https://www.mdpi.com/2225-1146/7/2/26

 

 

3.     Going back to the intro, where the author explains about journal policies, the author seems unaware that the recent movement was started by Basic and Applied Social Psychology  (BASP). In fact, at a conference, Wasserstein (who the author cited) credited BASP with “kickstarting” the American Statistical Association and even showed a slide of the editorial! Here is the reference.

 

Trafimow, D., & Marks, M. (2015). Editorial. Basic and Applied Social Psychology, 37(1), 1-2.

Doi: 10.1080/01973533.2015.1012991

 

In summary, although I have suggested changes, none of these should be too much of a barrier for the author. Therefore, I hope and expect that a revision should traverse much of the distance towards eventual publication.

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

This is a well-written manuscript on a long-discussed topic.  Nothing is novel but given the controversary over P-values and tests of significance, revisiting these issues could be helpful, especially in a more general sciences journal like this one.  

Comments:

1. I understand the use of a simple pre/post test for the examples but in general a single arm trial is not a preferred design.  It's better to also include a control group.  I'm not suggesting this be added as it doesn't necessarily remedy the issue you raise.  I just think some mention of the design not being the best choice would be helpful to the readers. 

2. Should a one-sided or two-sided test be performed here?  Given the subject matter, I could see an argument for a one-sided test.  A one-sided test would alter some statistical results.  

3. The second example is not an example that I would trust the inference from a paired t-test (i.e., conditions for inference are not satisfied).  In fact, if I perform a Fisher's permutation test, the two-sided P-value comes back smaller than 0.05.  Because of this, I don't think this example is a good one.  It suggests the researcher is not looking at the data (to assess the validity of inference) as well as not interpreting the data properly.   

Typos

1. Page 2 line 69  currently have "they sees"

2. Page 8 line 178 I find the beginning of that paragraph awkward to read.  I'd drop the "in this example"

3. You seem to swtich between data being plural (pg 10 line 294) and singular (pg 10 line 292).  I'd prefer to always see it plural but at the very least you should be consistent throughout.  Please look through the manuscript as there are numerous places where this arises.  

4.  pg 9 line 238 currently have "representativeness o the sample"

5. pg 9 line 247  should have "p-value describes"

6 pg 11 line 344.  I don't think you need "that" at the start of the line 

       

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The author has responded reasonably to my suggestions. Therefore, I think it is fine to proceed to the next stage of the publication process. It only remains to congratulate the author for responding well to what I hope were constructive criticisms on the earlier version. 

Nice work! David Trafimow