Open Access This article is
- freely available
Nutrients 2019, 11(5), 1126; https://doi.org/10.3390/nu11051126
The Implementation of Randomization Requires Corrected Analyses. Comment on “Comprehensive Nutritional and Dietary Intervention for Autism Spectrum Disorder—A Randomized, Controlled 12-Month Trial, Nutrients 2018, 10, 369”
Department of Applied Health Science, School of Public Health - Bloomington, Indiana University, Bloomington, IN 47405, USA
Department of Epidemiology and Biostatistics, School of Public Health - Bloomington, Indiana University, Bloomington, IN 47405, USA
Department of Statistics, Columbia University, New York, NY 10027, USA
Authors to whom correspondence should be addressed.
Received: 23 March 2019 / Accepted: 16 May 2019 / Published: 21 May 2019
We commend Adams et al.  for undertaking a lengthy intervention with a randomized design. However, we have concerns about the implementation of the randomization and analysis of results.
Using block randomization, the authors enrolled 37 participants to the treatment and 30 to the non-treatment groups. In response to our email asking how block randomization led to these discrepant group sizes, the corresponding author added that they balanced participants between high and low “ABA therapy” and by age, and paired siblings to the same treatment groups, then randomly assigned within each batch. We thank the author for communicating these details; however, this raises three specific concerns regarding the randomization and/or the extent to which the analyses used are correct given the randomization.
First, the randomization actually used was not described in the published report. Specifically, the authors reported that participants “were enrolled on a rolling basis, and participants with similar ages were matched and then randomly assigned to one of the two groups” . However, the authors clarified in a personal communication that randomization also involved balancing by level of ABA therapy, and randomly assigning within each batch, which were not reported.
Second, given that the authors also communicated to us that siblings were assigned the same treatment, this means that the independence of observations assumptions of ordinary least squares analyses (which include the analyses the authors used) is violated [2,3], and analyses must be adjusted to account for this clustering effect . Observations within clusters are often correlated, and effects may be the result of other factors besides the treatment, such as the same family environment. As such, not accounting for clustering may cause understatement of uncertainty in the intervention effect, and analysis as a partially clustered design should be performed .
Third, although block randomization can be an appropriate design, block effects should be included in statistical analyses .
One additional concern was the curious duality that the authors utilized two-tailed t-tests for biomarker outcomes but one-tailed t-tests for behavioral outcomes based on a hypothesized directionality . One-tailed tests are often inappropriate because treatments could potentially worsen outcomes , and the authors even describe examples of behaviors worsening . Further, their previous study utilized two-tailed t-tests with several of the same behavioral measures . Because this study was retrospectively registered , we cannot confirm whether the choice of test was an a priori analysis decision.
Because the data are not publicly available, we are unable to perform the corrected analyses. We therefore request that the authors do so, and that they amend their report to accurately and completely describe their methods in the spirit of rigorous and transparent research . We offer our assistance conducting reanalysis if requested.
Supported in part by NIH grants R25DK099080 and R25HL124208. The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization.
Conflicts of Interest
In the last 12 months, Dr. Allison has received personal payments or promises for same from for-profit organizations including: Biofortis; Fish & Richardson, P.C.; IKEA; Law Offices of Ronald Marron; Sage Publishing; Tomasik, Kotin & Kasserman LLC; Nestle; WW (formerly Weight Watchers International, LLC). Dr. Allison is an unpaid member of the International Life Sciences Institute North America Board of Trustees. In the last 12 months, Dr. Brown has received personal payments or paid travel from: American Society for Nutrition, Indiana University, Kentuckiana Health Collaborative, Rippe Lifestyle Institute, Inc. Indiana University has received grants from the following entities to support some of the authors’ research or educational activities: NIH; Alliance for Potato Research and Education; American Federation for Aging Research; Dairy Management Inc; Herbalife; Laura and John Arnold Foundation; Oxford University Press; Sloan Foundation; University of Alabama at Birmingham. The other authors declare no conflicts of interest.
- Adams, J.B.; Audhya, T.; Geis, E.; Gehn, E.; Fimbres, V.; Pollard, E.L.; Mitchell, J.; Ingram, J.; Hellmers, R.; Laake, D.; et al. Comprehensive Nutritional and Dietary Intervention for Autism Spectrum Disorder-A Randomized, Controlled 12-Month Trial. Nutrients 2018, 10, 369. [Google Scholar] [CrossRef] [PubMed]
- Davenport, J.M. The Effects of Violations of Independence Assumptions in the One-Way ANOVA AU - Scariano, Stephen M. Am. Stat. 1987, 41, 123–129. [Google Scholar] [CrossRef]
- Kenny, D.A.; Judd, C.M. Consequences of violating the independence assumption in analysis of variance. Psychol. Bull. 1986, 99, 422–431. [Google Scholar] [CrossRef]
- Sainani, K. The importance of accounting for correlated observations. PMR 2010, 2, 858–861. [Google Scholar] [CrossRef] [PubMed]
- Baldwin, S.A.; Bauer, D.J.; Stice, E.; Rohde, P. Evaluating models for partially clustered designs. Psychol. Methods 2011, 16, 149–165. [Google Scholar] [CrossRef] [PubMed]
- Kahan, B.C.; Morris, T.P. Improper analysis of trials randomised using stratified blocks or minimisation. Stat. Med. 2012, 31, 328–340. [Google Scholar] [CrossRef] [PubMed]
- Streiner, D.L. Statistics Commentary Series: Commentary #12-One--Tailed and Two-Tailed Tests. J. Clin. Psychopharmacol. 2015, 35, 628–629. [Google Scholar] [CrossRef] [PubMed]
- Adams, J.B.; Audhya, T.; McDonough-Means, S.; Rubin, R.A.; Quig, D.; Geis, E.; Gehn, E.; Loresto, M.; Mitchell, J.; Atwood, S.; et al. Effect of a vitamin/mineral supplement on children and adults with autism. BMC Pediatr. 2011, 11, 111. [Google Scholar] [CrossRef] [PubMed]
- Nutritional and Dietary Treatment Study for Children/Adults With Autism. Available online: https://ClinicalTrials.gov/show/NCT02059577 (accessed on 24 January 2019).
- Brown, A.W.; Kaiser, K.A.; Allison, D.B. Issues with data and analyses: Errors, underlying themes, and potential solutions. Proc. Natl Acad. Sci. USA 2018, 115, 2563–2570. [Google Scholar] [CrossRef] [PubMed][Green Version]
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).