On p-Values and Statistical Significance
- (1)
- “p-values can indicate how incompatible the data are with a specified statistical model.” Based on the observed data, the researcher either rejects or “fails to reject” the null hypothesis.
- (2)
- “p-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.” They can only tell us how consistent the data are with the null hypothesis.
- (3)
- “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.” Decision-making must not rest solely on a p-value. Researchers should also consider a broad range of other factors, e.g., study design, data quality, related prior evidence, plausibility of mechanism, real-world costs and benefits, etc.
- (4)
- “Proper inference requires full reporting and transparency.” Researchers should not conduct multiple, unplanned analyses of the data and (selectively) report only those with certain p-values (i.e., those reaching statistical significance). This is usually called cherry-picking, data-dredging, significance-chasing, significance-questing, selective-inference, or p-hacking, and leads to a spurious excess of statistically significant results in the biomedical literature.
- (5)
- “A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.” For example, smaller p-values do not necessarily imply the presence of larger or more important effects, and larger p-values do not imply a lack of effect or lack of importance.
- (6)
- “By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.” Analysis of the data should not end with the calculation of a p-value. Depending on the context, p-values can be supplemented by other measures of evidence that can more directly address the effect size and its associated uncertainty (e.g., effect size estimates, confidence and prediction intervals, likelihood ratios, or graphical representations).
Author Contributions
Conflicts of Interest
References
- Wasserstein, R.; Lazar, N. The ASA’s Statement on p-Values: Context, Process, and Purpose. Am. Stat. 2016, 70, 129–133. [Google Scholar] [CrossRef] [Green Version]
- Fisher, R.A. The Statistical Method in Psychical Research. Proc. Soc. Psych. Res. 1929, 39, 189–192. [Google Scholar]
- Neyman, J.; Pearson, E.S. On the Problem of the Most Efficient Tests of Statistical Hypotheses. Philos. Trans. R. Soc. Lond. 1933, 231, 289–337. [Google Scholar]
- Wang, M.; Long, Q. Addressing Common Misuses and Pitfalls of p values in Biomedical Research. Cancer Res. 2022, 82, 2674–2677. [Google Scholar] [CrossRef] [PubMed]
- Wasserstein, R.L.; Schirm, A.L.; Lazar, N.A. Moving to a World Beyond “p < 0.05”. Am. Stat. 2019, 73, 1–19. [Google Scholar]
- Greenland, S.; Senn, S.J.; Rothman, K.J.; Carlin, J.B.; Poole, C.; Goodman, S.N.; Altman, D.G. Statistical Tests, p Values, Confidence Intervals, and Power: A Guide to Misinterpretations. Eur. J. Epidemiol. 2016, 31, 337–350. [Google Scholar] [CrossRef] [Green Version]
- Nuzzo, R. Scientific Method: Statistical Errors. Nature 2014, 506, 150–152. [Google Scholar] [CrossRef] [Green Version]
- Gelman, A.; Loken, E. The Statistical Crisis in Science. Am. Sci. 2014, 102, 460–465. [Google Scholar] [CrossRef]
- Ioannidis, J.P.A. Why Most Published Research Findings Are False. PLoS Med. 2005, 2, e124. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Rothman, K.J. A Show of Confidence. N. Engl. J. Med. 1978, 299, 1362–1363. [Google Scholar] [CrossRef]
- Piovani, D.; Bonovas, S. Real World—Big Data Analytics in Healthcare. Int. J. Environ. Res. Public Health 2022, 19, 11677. [Google Scholar] [CrossRef]
- Amrhein, V.; Greenland, S.; McShane, B. Scientists Rise Up Against Statistical Significance. Nature 2019, 567, 305–307. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Altman, D.G.; Bland, J.M. Absence of Evidence is Not Evidence of Absence. BMJ 1995, 311, 485. [Google Scholar] [CrossRef] [PubMed]
- Halsey, L.G.; Curran-Everett, D.; Vowler, S.L.; Drummond, G.B. The Fickle p Value Generates Irreproducible Results. Nat. Methods 2015, 12, 179–185. [Google Scholar] [CrossRef] [PubMed]
- Goodman, S. A Dirty Dozen: Twelve p-Value Misconceptions. Semin. Hematol. 2008, 45, 135–140. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Ioannidis, J.P.A. The Importance of Predefined Rules and Prespecified Statistical Analyses: Do Not Abandon Significance. JAMA 2019, 321, 2067–2068. [Google Scholar] [CrossRef] [PubMed]
- Simera, I.; Moher, D.; Hirst, A.; Hoey, J.; Schulz, K.F.; Altman, D.G. Transparent and Accurate Reporting Increases Reliability, Utility, and Impact of Your Research: Reporting Guidelines and the EQUATOR Network. BMC Med. 2010, 8, 24. [Google Scholar] [CrossRef] [PubMed]
Treatment Group | |||
---|---|---|---|
Deaths | Active Drug | Placebo | Total |
Yes | 28 | 40 | 68 |
No | 172 | 160 | 332 |
Total | 200 | 200 | 400 |
Proportion of deaths | 14% | 20% | 17% |
Treatment Group | |||
---|---|---|---|
Deaths | Active Drug | Placebo | Total |
Yes | 119 | 170 | 289 |
No | 731 | 680 | 1411 |
Total | 850 | 850 | 1700 |
Proportion of deaths | 14% | 20% | 17% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Bonovas, S.; Piovani, D. On p-Values and Statistical Significance. J. Clin. Med. 2023, 12, 900. https://doi.org/10.3390/jcm12030900
Bonovas S, Piovani D. On p-Values and Statistical Significance. Journal of Clinical Medicine. 2023; 12(3):900. https://doi.org/10.3390/jcm12030900
Chicago/Turabian StyleBonovas, Stefanos, and Daniele Piovani. 2023. "On p-Values and Statistical Significance" Journal of Clinical Medicine 12, no. 3: 900. https://doi.org/10.3390/jcm12030900