Next Article in Journal
A Reply to Contrary to the Conclusions Stated in the Paper, Only Dry Fat-Free Mass Was Different between Groups upon Reanalysis—Comment on: “Intermittent Energy Restriction Attenuates the Loss of Fat-Free Mass in Resistance Trained Individuals: A Randomized Controlled Trial”
Previous Article in Journal
Does Warm-up Type Matter? A Comparison between Traditional and Functional Inertial Warm-up in Young Soccer Players
Previous Article in Special Issue
Intermittent Energy Restriction Attenuates the Loss of Fat Free Mass in Resistance Trained Individuals. A Randomized Controlled Trial
 
 
Reply published on 24 November 2020, see J. Funct. Morphol. Kinesiol. 2020, 5(4), 86.
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Comment

Contrary to the Conclusions Stated in the Paper, Only Dry Fat-Free Mass Was Different between Groups upon Reanalysis. Comment on: “Intermittent Energy Restriction Attenuates the Loss of Fat-Free Mass in Resistance Trained Individuals. A Randomized Controlled Trial”

1
Faculty of Science, School of Human Sciences, The University of Western Australia, Crawley, WA 6009, Australia
2
Department of Applied Health Science, School of Public Health-Bloomington, Indiana University, Bloomington, IN 47405, USA
3
Department of Epidemiology and Biostatistics, School of Public Health-Bloomington, Indiana University, Bloomington, IN 47405, USA
*
Authors to whom correspondence should be addressed.
J. Funct. Morphol. Kinesiol. 2020, 5(4), 85; https://doi.org/10.3390/jfmk5040085
Received: 1 September 2020 / Accepted: 11 November 2020 / Published: 20 November 2020
(This article belongs to the Special Issue Research on Sports Nutrition: Body Composition and Performance)
Campbell and colleagues recently published a randomised controlled trial investigating the effects of diets involving intermittent energy restriction versus continuous energy restriction on changes in body composition and resting metabolic rate (RMR) in resistance-trained adults [1]. Intermittent energy restriction is a topic of great interest to athletes, and this was a greatly needed study, because previous research in the realm of intermittent energy restriction has focused on populations with overweight or obesity. Moreover, the diet that Campbell and colleagues tested in their trial (7 weeks with 5 days per week of prescribed moderate energy restriction and 2 weekend days per week with no prescribed energy restriction) is of relevance to many athletes who implement weekend ‘refeeds’ or ‘diet breaks’ during energy restriction regimes in an effort to prevent loss of fat-free mass (FFM) and reductions in RMR [2]. In keeping with the hypothesised effects of intermittent energy restriction, the authors concluded that “a 2-day carbohydrate refeed preserves FFM, dry FFM and RMR during energy restriction compared to continuous energy restriction in RT (resistance-trained) individuals.” However, we have two concerns regarding the statistical analysis and interpretation of the data, and these concerns lead us to believe that re-evaluation of the authors’ conclusion is warranted. Our first concern is that the authors have drawn conclusions from their data based on differences in nominal significance (i.e., the “DINS error”, detailed below), rather than on actual differences between the diet groups. Our second concern is that the authors have based their conclusion on analysis of only those people who completed the trial (a completers-only analysis, also known as a per-protocol analysis), rather than also analysing all people who were randomised to the diets (intention-to-treat analysis, also detailed below).
The DINS error in randomised controlled trials occurs when authors wrongly conclude that the randomised group results demonstrate an effect of treatment assignment when one group shows a significant change from baseline, but another group does not. For example, Campbell et al. conclude that the intermittent/refeed diet results in greater retention of FFM and RMR than the continuous diet, presumably because there were statistically significant reductions in FFM and RMR relative to baseline (pre-diet) in the continuous diet group, but no statistically significant pre-diet to post-diet changes in the intermittent/refeed diet group, as reported in their Table 3. In other words, they appear to have interpreted the significant within-group decreases from baseline in FFM and RMR in the continuous diet group (but not in the intermittent/refeed diet group) as evidence of a significant between-group difference. This analytic strategy is invalid, as noted previously in the literature [3,4,5]. The DINS error can increase the chance of false positive results (i.e., concluding that there is a difference between groups when in fact no difference exists) to up to 50% (and even higher if group sample sizes are unequal), compared to the usually-accepted false positive rate of 5% (as is the case when statistical significance is set at p < 0.05) [4,5,6]. Multiple papers using this invalid analytic strategy have had to be corrected [7,8,9] or retracted [10,11].
In parallel-group randomised controlled trials, the correct way to determine whether treatment effects are different from each other is to directly compare outcomes in the treatment and control groups [3,12]. Campbell et al. undertook these direct comparisons between the intermittent/refeed diet and the continuous diet groups using a two group × two time point between–within factorial analysis of variance (ANOVA) with repeated measures. Contrary to their conclusion, there was no significant difference between diet groups for the change in FFM or RMR, albeit they did report a significantly greater preservation of dry FFM in the intermittent/refeed diet group. To investigate this further, we analysed the data provided in the publication’s online supplementary material using analysis of covariance (ANCOVA) with baseline values as a covariate, which has been shown to generally provide greater power than ANOVA in randomised controlled trials [13]. Like Campbell et al., we also found no significant difference between the intermittent/refeed and continuous diet groups for the change in FFM or RMR, albeit we did also observe significantly greater retention of dry FFM upon completion of the intermittent/refeed diet versus the continuous diet (p = 0.0004).
In addition to the DINS error, the paper by Campbell et al. does not report any intention-to-treat analysis, instead analysing data only from participants who completed the 7-week diets (a completers-only analysis). Considering that 27 of 54 (50% of) participants that underwent baseline testing completed the diet and trial as intended, those that withdrew or were excluded from the trial represent a large proportion of the original sample size. Completers-only analyses have been shown to overestimate the effects of interventions and have led to retractions [14,15], because they only examine those participants who remained in the trial, with these participants being more likely to have adhered to the intervention than those who withdrew from the trial. This is an important point, because non-compliance and withdrawal from interventions is likely to happen in actual practice [16]. To investigate the possibility that the completers-only analyses conducted by Campbell et al. may have overestimated treatment effects, we undertook an intention-to-treat analysis on the three outcomes mentioned above—FFM, dry FFM, and RMR (see our calculations at https://doi.org/10.5281/zenodo.3961834), using the Baseline Observation Carried Forward (BOCF) approach. BOCF is less robust than approaches such as multiple imputation, but was the only approach available to us given lack of data on the 27 of 54 participants that withdrew or were excluded from the trial. We conducted our analysis on changes from baseline, assuming that missing participants had no change from baseline [17]. Specifically, we compared the two diets groups in terms of the change from baseline in each of the three outcomes, using a between-group Welch’s t-test. We used a Welch’s t-test rather than a Student’s t-test, because the Welch’s t-test does not assume normality of the data (unlike a Student’s t-test), and with half of the data set having a value of zero (i.e., no change from baseline), the data were unlikely to be normally distributed. It has been argued that the Welch’s t-test should be used by default rather than Student’s t-test [18]. For FFM and RMR, we saw no significant differences between diet groups in change scores (FFM p values for Welch’s t-test on completers-only and BOCF data of 0.1795 and 0.1656, respectively, and RMR p values of 0.4916 and 0.4452, respectively). For dry FFM, we again detected a significant between-group difference, but the magnitude of the difference between diet groups was smaller with the BOCF analysis than with the completers-only analysis. Specifically, for the BOCF analysis, dry FFM was significantly greater in the intermittent/refeed diet group compared to the continuous diet group, by 0.9 kg (95% confidence interval 0.33 to 1.46 kg; p = 0.0028), which is about half the estimate from the completers-only analysis, where the difference was 1.7 kg (95% confidence interval 0.86 to 2.57 kg; p = 0.0004).
We commend the authors for sharing their data as a supplement in the spirit of more transparent and reproducible science. Their sharing of data enabled us to confirm that the effects of intermittent versus continuous energy restriction on dry FFM in resistance-trained adults seem to be robust to completers-only ANCOVA, completers-only Welch’s t-test on change scores, and to a simplistic (BOCF) intention-to-treat analysis, also using Welch’s t-test on change scores. Although the latter changed the magnitude of the estimated effect size, the p value was less than a Bonferroni-corrected alpha for multiple outcomes, demonstrating statistical significance. Conversely, we were able to determine that the comparison of within-group changes using the DINS approach resulted in erroneous conclusions about the effects of an intermittent/refeed diet versus a continuous diet on FFM and RMR. Thus, the original conclusions of Campbell et al. should be corrected to indicate that only one of these outcomes (dry FFM) was significantly different between the intermittent/refeed and continuous diets. Upon correction, we also request that p values be reported exactly, as per statistical reporting guidelines [19] (i.e., provide exact p values instead of statements such as p < 0.05), which will further aid readers in interpretation of the data.

Acknowledgments

We thank Greyson Foote with the Indiana University School of Public Health-Bloomington Biostatistics Consulting Center for verifying the BOCF code. JJP is supported by the Australian Department of Education and Training (DET) via a Research Training Program Scholarship. AWB and DBA are supported in part by the National Heart, Lung, and Blood Institute and by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health (NIH), under Award Numbers R25HL124208 and R25DK090880. CJV is supported by the Gordon and Betty Moore Foundation. AS is supported by the National Health and Medical Research Council (NHMRC) of Australia via a Senior Research Fellowship (1135897). The content of this work is solely the responsibility of the authors and does not necessarily represent the official views of the NIH, the DET, the NHMRC, or any other organization.

Disclosures

JJP reports no disclosures. In the year prior to submission, AWB received speaking fees from Purdue University; consulting fees from LA NORC as well as from the Pennington Biomedical Research Center; and grants through his institution from Alliance for Potato Research & Education, National Cattlemen’s Beef Association, NIH/NHLBI, and NIH/NIDDK. In addition, he has been involved in research for which his institution or colleagues have received grants or contracts from the Gordon and Betty Moore Foundation, Indiana CTSI, the National Cattlemen’s Beef Association, NIH/NHLBI, NIH/NIA, and the Sloan Foundation. His wife is employed by Reckitt Benckiser. In the past 12 months, DBA has received personal payments or promises for same from: Alkermes, Inc., American Society for Nutrition; Biofortis; California Walnut Commission; Henry Stewart Talks; Indiana University; Johns Hopkins University; Law Offices of Ronald Marron; Medpace; Sage Publishing; The Obesity Society; Tomasik, Kotin & Kasserman LLC; University of Alabama at Birmingham; University of Miami. Donations to a foundation have been made on his behalf by the Northarvest Bean Growers Association. DBA has previously served as an unpaid member of the International Life Sciences Institute North America Board of Trustees. DBA’s institution, Indiana University, and the Indiana University Foundation have received funds to support his research or educational activities from: NIH; Eli Lilly, Alliance for Potato Research and Education; American Federation for Aging Research; Dairy Management Inc.; Herbalife; Laura and John Arnold Foundation; Mars, Inc., National Cattlemen’s Beef Association, Oxford University Press, the Sloan Foundation, The Gordon and Betty Moore Foundation, and numerous other for-profit and non-profit organizations and private donors to support the work of the School of Public Health more broadly. CV reports no disclosures. AS owns 50% of the shares in Zuman International, which receives royalties for books she has written about adult weight management and payments for presentations at industry conferences. She has also received presentation fees and travel reimbursements from Eli Lilly and Co, the Pharmacy Guild of Australia, Novo Nordisk, the Dietitians Association of Australia, Shoalhaven Family Medical Centres, the Pharmaceutical Society of Australia, and Metagenics, and served on the Nestlé Health Science Optifast VLCD advisory board from 2016 to 2018.

References

  1. Campbell, B.; Aguilar, D.; Colenso-Semple, L.M.; Hartke, K.; Fleming, A.R.; Fox, C.D.; Longstrom, J.M.; Rogers, G.E.; Mathas, D.B.; Wong, V.; et al. Intermittent Energy Restriction Attenuates the Loss of Fat Free Mass in Resistance Trained Individuals. A Randomized Controlled Trial. J. Funct. Morphol. Kinesiol. 2020, 5, 19. [Google Scholar] [CrossRef][Green Version]
  2. Peos, J.J.; Norton, L.E.; Helms, E.R.; Galpin, A.J.; Fournier, P.A. Intermittent Dieting: Theoretical Considerations for the Athlete. Sports 2019, 7, 22. [Google Scholar] [CrossRef][Green Version]
  3. Allison, D.B.; Brown, A.W.; George, B.J.; Kaiser, K.A. Reproducibility: A tragedy of errors. Nat. Cell Biol. 2016, 530, 27–29. [Google Scholar] [CrossRef][Green Version]
  4. Bland, M.; Altman, D.G. Comparisons against baseline within randomised groups are often used and can be highly misleading. Trials 2011, 12, 264. [Google Scholar] [CrossRef][Green Version]
  5. Bland, M.; Altman, D.G. Best (but oft forgotten) practices: Testing for treatment effects in randomized trials by separate analyses of changes from baseline in each group is a misleading approach. Am. J. Clin. Nutr. 2015, 102, 991–994. [Google Scholar] [CrossRef]
  6. George, B.J.; Beasley, T.M.; Brown, A.W.; Dawson, J.; Dimova, R.; Divers, J.; Goldsby, T.U.; Heo, M.; Kaiser, K.A.; Keith, S.W.; et al. Common scientific and statistical errors in obesity research. Obesity 2016, 24, 781–790. [Google Scholar] [CrossRef]
  7. Allison, D.B. RE: Statistical Interpretation Error in Metformin Trial Article. Pediatrics 2017, 140, e20173231A. [Google Scholar] [CrossRef][Green Version]
  8. Allison, D.B. The Conclusions Are Unsupported by the Data, Are Based on Invalid Analyses, Are Incorrect, and Should be Corrected: Letter Regarding “Sleep Quality and Body Composition Variations in Obese Male Adults after 14 weeks of Yoga Intervention: A Randomized Controlled Trial”. Int. J. Yoga 2018, 11, 83–84. [Google Scholar]
  9. Allison, D.B.; Antoine, L.H.; George, B.J. Incorrect statistical method in parallel-groups RCT led to unsubstantiated conclusions. Lipids Health Dis. 2016, 15, 1–5. [Google Scholar] [CrossRef][Green Version]
  10. Cassani, R.S.L.; Fassini, P.G.; Silvah, J.H.; Lima, C.M.M.; Marchini, J.S. Impact of weight loss diet associated with flaxseed on inflammatory markers in men with cardiovascular risk factors: A clinical study. Nutr. J. 2015, 14, 5. [Google Scholar] [CrossRef]
  11. Dimova, R.B.; Allison, D.B. Inappropriate statistical method in a parallel-group randomized controlled trial results in unsubstantiated conclusions. Nutr. J. 2016, 15, 58. [Google Scholar] [CrossRef][Green Version]
  12. Moher, D.; Hopewell, S.; Schulz, K.F.; Montori, V.; Gøtzsche, P.C.; Devereaux, P.J.; Elbourne, D.; Egger, M.; Altman, D.G. CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomised trials. Br. J. Med. 2010, 340, c869. [Google Scholar] [CrossRef][Green Version]
  13. Van Breukelen, G. ANCOVA versus change from baseline had more power in randomized studies and more bias in nonrandomized studies. J. Clin. Epidemiol. 2006, 59, 920–925. [Google Scholar] [CrossRef]
  14. Albada, A.; Van Dulmen, S.; Bensing, J.M.; Ausems, M.G.E.M. Retraction: Effects of a pre-visit educational website on information recall and needs fulfilment in breast cancer genetic counselling, a randomized controlled trial. Breast Cancer Res. 2012, 14, 402. [Google Scholar] [CrossRef][Green Version]
  15. Karp, D.D.; Paz-Ares, L.G.; Novello, S.; Haluska, P.; Garland, L.; Cardenal, F.; Blakely, L.J.; Eisenberg, P.D.; Langer, C.J.; Blumenschein, G.; et al. Phase II Study of the Anti–Insulin-Like Growth Factor Type 1 Receptor Antibody CP-751,871 in Combination With Paclitaxel and Carboplatin in Previously Untreated, Locally Advanced, or Metastatic Non–Small-Cell Lung Cancer. J. Clin. Oncol. 2009, 27, 2516–2522. [Google Scholar] [CrossRef]
  16. Heritier, S.R.; Gebski, V.J.; Keech, A.C. Inclusion of patients in clinical trial analysis: The intention-to-treat principle. Med J. Aust. 2003, 179, 438–440. [Google Scholar] [CrossRef]
  17. Kaiser, K.A.; Affuso, O.; Beasley, T.M.; Allison, D.B. Getting carried away: A note showing baseline observation carried forward (BOCF) results can be calculated from published complete-cases results. Int. J. Obes. 2011, 36, 886–889. [Google Scholar] [CrossRef][Green Version]
  18. Delacre, M.; Lakens, D.D.; Leys, C. Why Psychologists Should by Default Use Welch’s t-test Instead of Student’s t-test. Int. Rev. Soc. Psychol. 2017, 30, 92. [Google Scholar] [CrossRef][Green Version]
  19. Lang, T.A.; Altman, D.G. Basic statistical reporting for articles published in Biomedical Journals: The “Statistical Analyses and Methods in the Published Literature” or the SAMPL Guidelines. Int. J. Nurs. Stud. 2015, 52, 5–9. [Google Scholar] [CrossRef]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Peos, J.; Brown, A.W.; Vorland, C.J.; Allison, D.B.; Sainsbury, A. Contrary to the Conclusions Stated in the Paper, Only Dry Fat-Free Mass Was Different between Groups upon Reanalysis. Comment on: “Intermittent Energy Restriction Attenuates the Loss of Fat-Free Mass in Resistance Trained Individuals. A Randomized Controlled Trial”. J. Funct. Morphol. Kinesiol. 2020, 5, 85. https://doi.org/10.3390/jfmk5040085

AMA Style

Peos J, Brown AW, Vorland CJ, Allison DB, Sainsbury A. Contrary to the Conclusions Stated in the Paper, Only Dry Fat-Free Mass Was Different between Groups upon Reanalysis. Comment on: “Intermittent Energy Restriction Attenuates the Loss of Fat-Free Mass in Resistance Trained Individuals. A Randomized Controlled Trial”. Journal of Functional Morphology and Kinesiology. 2020; 5(4):85. https://doi.org/10.3390/jfmk5040085

Chicago/Turabian Style

Peos, Jackson, Andrew W. Brown, Colby J. Vorland, David B. Allison, and Amanda Sainsbury. 2020. "Contrary to the Conclusions Stated in the Paper, Only Dry Fat-Free Mass Was Different between Groups upon Reanalysis. Comment on: “Intermittent Energy Restriction Attenuates the Loss of Fat-Free Mass in Resistance Trained Individuals. A Randomized Controlled Trial”" Journal of Functional Morphology and Kinesiology 5, no. 4: 85. https://doi.org/10.3390/jfmk5040085

Article Metrics

Back to TopTop