We read with interest the publication by Lightowler et al., who concluded that the inclusion of isomaltulose in the context of an energy-reduced diet reduced weight and fat mass to a greater extent than sucrose [1
]. We praise the authors for their use of blinding, randomization, and power calculation in their study to enhance the rigor of the experiment. Unfortunately, there are errors in the statistical conduct and interpretation that do not support these conclusions.
In the abstract, the authors write: “During the 12 weeks, both groups significantly lost weight (p
< 0.001), which was more pronounced following [isomaltulose] (−3.2 ± 2.9 vs. −2.1 ± 2.6 kg; p = 0.258
] [emphasis added
]. The discussion also notes that “…consumption of [isomaltulose] compared to that of [sucrose] was more effective at promoting weight loss”. The authors stated in their methods that statistical significance was set at p
< 0.05, so by the authors’ own designation, the effect on weight loss of isomaltulose was not statistically different from that of sucrose. Highlighting a beneficial effect despite a nonsignificant difference is a reporting strategy that is classified as “spin” [2
Furthermore, throughout the paper, some conclusions about the differences between groups are made from within-group instead of between-group statistical tests, which is called a differences in nominal significance (DINS) error [3
] and results in inflated type 1 error rates [4
]. An example of this appears in the abstract: “Moreover, for participants in the [isomaltulose] group, this was accompanied by a significant reduction in fat mass ([isomaltulose]: −1.9 ± 2.5, p
= 0.005; [sucrose]: −0.9 ± 2.6%, p
= 0.224)” [1
]. In the results, the between-group p
-value is noted as p
= 0.169, again not meeting the authors’ declared threshold for the appropriate between-group comparison.
Although there is much debate about whether to use a cutoff for statistical significance [6
], the two examples discussed above do not provide strong evidence that these data are incompatible with the model under the null hypothesis of there being no difference between the groups. It is also inappropriate to solely use the differences between sample means to declare a meaningful difference between groups without accounting for the variability in those estimates [8
Fortunately, the errors we address herein are not from the structure of the study and can therefore be easily addressed by clarifying the interpretation of the results with a corrigendum.
Supported in part by the Gordon and Betty Moore Foundation and NIH grant R25HL124208. The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization.
Conflicts of Interest
In the past 36 months, Brown has received travel expenses from the Academy of Nutrition and Dietetics, National Academy of Sciences, University of Louisville, and University of Michigan; speaking fees from the Academy of Nutrition and Dietetics, Kentuckiana Health Collaborative, Purdue University, and Rippe Lifestyle Institute, Inc.; consulting fees from Epigeum (Oxford University Press) and LA NORC; and grants through his institution from Dairy Management, Inc., the National Cattlemen’s Beef Association, and NIH/NIGMS-NIA-NINDS. He has been involved in research for which his institution or colleagues have received grants from Dairy Management, Inc., the Gordon and Betty Moore Foundation, Indiana CTSI (Bloomington), the National Cattlemen’s Beef Association, NIH/NHLBI, NIH/NIA, NIH/NIDDK, NIH/OD, and the Sloan Foundation. In the past 36 months, Kyle has received consulting fees from Novo Nordisk and Tivity Health. Vorland reports no disclosures.
- Lightowler, H.; Schweitzer, L.; Theis, S.; Henry, C.J. Changes in Weight and Substrate Oxidation in Overweight Adults Following Isomaltulose Intake During a 12-Week Weight Loss Intervention: A Randomized, Double-Blind, Controlled Trial. Nutrients 2019, 11, 2367. [Google Scholar] [CrossRef] [PubMed]
- Boutron, I.; Dutton, S.; Ravaud, P.; Altman, D.G. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. Jama 2010, 303, 2058–2064. [Google Scholar] [CrossRef] [PubMed]
- Allison, D.B.; Brown, A.W.; George, B.J.; Kaiser, K.A. Reproducibility: A tragedy of errors. Nature 2016, 530, 27–29. [Google Scholar] [CrossRef] [PubMed]
- Bland, J.M.; Altman, D.G. Comparisons against baseline within randomised groups are often used and can be highly misleading. Trials 2011, 12, 264. [Google Scholar] [CrossRef] [PubMed]
- Bland, J.M.; Altman, D.G. Best (but oft forgotten) practices: Testing for treatment effects in randomized trials by separate analyses of changes from baseline in each group is a misleading approach. Am. J. Clin. Nutr. 2015, 102, 991–994. [Google Scholar] [CrossRef] [PubMed]
- Wasserstein, R.L.; Lazar, N.A. The ASA Statement on p-Values: Context, Process, and Purpose. Am. Stat. 2016, 70, 129–133. [Google Scholar] [CrossRef]
- Amrhein, V.; Greenland, S.; McShane, B. Scientists Rise Up Against Statistical Significance. Nature 2019, 305–307, 567. [Google Scholar] [CrossRef] [PubMed]
- Brown, A.W.; Altman, D.G.; Baranowski, T.; Bland, J.M.; Dawson, J.A.; Dhurandhar, N.V.; Dowla, S.; Fontaine, K.R.; Gelman, A.; Heymsfield, S.B. Childhood obesity intervention studies: A narrative review and guide for investigators, authors, editors, reviewers, journalists, and readers to guard against exaggerated effectiveness claims. Obes. Rev. 2019, 20, 1523–1541. [Google Scholar] [CrossRef] [PubMed]
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).