# A Multilevel Bayesian Approach to Improve Effect Size Estimation in Regression Modeling of Metabolomics Data Utilizing Imputation with Uncertainty

^{1}

^{2}

^{3}

^{4}

^{5}

^{6}

^{7}

^{8}

^{9}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

## 2. Results

#### 2.1. A Multilevel Bayesian Approach Has Higher Power, Controls for False Discovery, and More Reliably Estimates Metabolite Effect Size across a Variety of Simulated Scenarios

^{1}H-NMR metabolomics data from two outcome groups of patients with septic shock (survivors and non-survivors). For both groups, we generated data from a multivariate normal distribution with the same means and covariance as the original data. We altered the sample size per group and the fraction of metabolites that are ‘truly’ different based on patient mortality. Two hundred-simulations for unique combinations of sample size per group and fraction of truly associated metabolites were completed (Figure 1).

#### 2.2. Imputation Incorporating Uncertainty Improves Predicted Metabolite Concentration

#### 2.3. Multilevel Bayesian Models Incorporating Uncertainty into Imputation Leads to Improving Effect Size Estimation in the Presence of Missing Data

#### 2.4. Application of Bayesian Models to Metabolomics Data

#### 2.5. Prior Probability Distribution Sensitivity Analysis

## 3. Discussion

_{βx}regularization term to be too small. Indeed, we see similar trends and overly conservative estimation in Bayesian estimated effect size in our simulations at low values for the fraction of significant metabolites (Supplementary Figure S2). However, our simulations also show that standard regression methods tend to inflate the AER, which agrees with previous reports that ‘statistically significant’ results tend to be upwardly biased in nature [3,4]. In this instance, alternative priors such as fixing ${\nu}_{\beta x}$ at a small value or introducing a lower bound for ${\sigma}_{\beta x}$ can help combat over shrinkage (Supplementary Figure S4). In the real GC-MS data, introducing a lower bound of 0.1 for ${\sigma}_{\beta x}$ identified a single metabolite related to ARDS, which is consistent with the Bonferroni or B-H FDR correction upon standard regression. More research will be needed to develop more effective prior distributions to prevent overcorrection and to assess model performance on untargeted metabolomic platforms.

## 4. Materials and Methods

#### 4.1. Simulation Approach

#### 4.1.1. Parameter Learning from Experimental Data

^{1}H-NMR as previously described [19,20,21]. All cause 90-day mortality was 122/228 (53.5%) and 106/228 (46.5%) survived beyond 90 days. After filtering metabolites with a missing rate less than 30%, there were 27 metabolites remaining. Concentrations were log transformed and each metabolite was scaled to have a mean of zero and a standard deviation of one.

#### 4.1.2. Simulation Parameters

#### 4.2. Multilevel Bayesian Logistic Regression Model

#### 4.3. Two-Stage Imputation Model

#### 4.3.1. Imputation Model

#### 4.3.2. Logistic Regression with Uncertainty

#### 4.3.3. Method Comparison

#### 4.4. Imputation Quality Evaluation

#### 4.5. Real Data Comparison

## 5. Conclusions

## Supplementary Materials

## Author Contributions

## Funding

## Conflicts of Interest

## Data and Code Availability

## References

- Baker, M. 1500 scientists lift the lid on reproducibility. Nature
**2016**, 533, 452–454. [Google Scholar] [CrossRef] [Green Version] - Head, M.L.; Holman, L.; Lanfear, R.; Kahn, A.T.; Jennions, M.D. The extent and consequences of p-hacking in science. PLOS Biol.
**2015**, 13, e1002106. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Amrhein, V.; Greenland, S.; McShane, B. Scientists rise up against statistical significance. Nature
**2019**, 567, 305–307. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Xiao, R.; Boehnke, M. Quantifying and correcting for the winner’s curse in genetic association studies. Genet. Epidemiol.
**2009**, 33, 453–462. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Gelman, A.; Carlin, J. Beyond power calculations: Assessing Type S (Sign) and Type M (Magnitude) errors. Perspect. Psychol. Sci. A J. Assoc. Psychol. Sci.
**2014**, 9, 641–651. [Google Scholar] [CrossRef] [PubMed] - Gelman, A.; Tuerlinckx, F. Type S error rates for classical and Bayesian single and multiple comparison procedures. Comput. Stat.
**2000**, 15, 373–390. [Google Scholar] [CrossRef] - Calin-Jageman, R.J.; Cumming, G. The new statistics for better science: Ask how much, how uncertain, and what else is known. Am. Stat.
**2019**, 73, 271–280. [Google Scholar] [CrossRef] [Green Version] - Cumming, G. The new statistics: Why and how. Psychol. Sci.
**2013**, 25, 7–29. [Google Scholar] [CrossRef] - Siskos, A.P.; Jain, P.; Römisch-Margl, W.; Bennett, M.; Achaintre, D.; Asad, Y.; Marney, L.; Richardson, L.; Koulman, A.; Griffin, J.L.; et al. Interlaboratory reproducibility of a targeted metabolomics platform for analysis of human serum and plasma. Anal. Chem.
**2017**, 89, 656–665. [Google Scholar] [CrossRef] - Li-Gao, R.; Hughes, D.A.; le Cessie, S.; de Mutsert, R.; den Heijer, M.; Rosendaal, F.R.; Willems van Dijk, K.; Timpson, N.J.; Mook-Kanamori, D.O. Assessment of reproducibility and biological variability of fasting and postprandial plasma metabolite concentrations using 1H NMR spectroscopy. PLoS ONE
**2019**, 14, e0218549. [Google Scholar] [CrossRef] [Green Version] - Chong, J.; Wishart, D.S.; Xia, J. Using MetaboAnalyst 4.0 for comprehensive and integrative metabolomics data analysis. Curr. Protoc. Bioinform.
**2019**, 68, e86. [Google Scholar] [CrossRef] [PubMed] - Gross, T.; Mapstone, M.; Miramontes, R.; Padilla, R.; Cheema, A.K.; Macciardi, F.; Federoff, H.J.; Fiandaca, M.S. Toward reproducible results from targeted metabolomic studies: Perspectives for data pre-processing and a basis for analytic pipeline development. Curr. Top. Med. Chem.
**2018**, 18, 883–895. [Google Scholar] [CrossRef] [PubMed] - Antonelli, J.; Claggett, B.L.; Henglin, M.; Kim, A.; Ovsak, G.; Kim, N.; Deng, K.; Rao, K.; Tyagi, O.; Watrous, J.D.; et al. Statistical workflow for feature selection in human metabolomics data. Metabolites
**2019**, 9, 143. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Benjamini, Y.; Hochberg, Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B
**1995**, 57, 289–300. [Google Scholar] [CrossRef] - Kruschke, J.K.; Liddell, T.M. The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychon. Bull. Rev.
**2018**, 25, 178–206. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Gelman, A.; Hill, J.; Yajima, M. Why we (usually) don’t have to worry about multiple comparisons. J. Res. Educ. Eff.
**2012**, 5, 189–211. [Google Scholar] [CrossRef] [Green Version] - Shah, J.; Brock, G.N.; Gaskins, J. BayesMetab: Treatment of missing values in metabolomic studies using a Bayesian modeling approach. BMC Bioinform.
**2019**, 20, 673. [Google Scholar] [CrossRef] - Wei, R.; Wang, J.; Jia, E.; Chen, T.; Ni, Y.; Jia, W. GSimp: A Gibbs sampler based left-censored missing value imputation approach for metabolomics studies. PLOS Comput. Biol.
**2018**, 14, e1005973. [Google Scholar] [CrossRef] [Green Version] - Puskarich, M.; Evans, C.; Karnovsky, A.; Gillies, C.; Jennaro, T.; Jones, A.; Stringer, K.A. Pretreatment Acetyl-Carnitine Levels Predict Mortality Benefit from L-Carnitine Treatment in Sepsis: A Pharmacometabolomics Based Clinical Trial Enrichment Strategy. In A103. SEPSIS: TRANSLATIONAL STUDIES; American Thoracic Society: New York, NY, USA, 2020; p. A2604. [Google Scholar] [CrossRef]
- McHugh, C.E.; Flott, T.L.; Schooff, C.R.; Smiley, Z.; Puskarich, M.A.; Myers, D.D.; Younger, J.G.; Jones, A.E.; Stringer, K.A. Rapid, reproducible, quantifiable NMR metabolomics: Methanol and methanol: Chloroform precipitation for removal of macromolecules in serum and whole blood. Metabolites
**2018**, 8, 93. [Google Scholar] [CrossRef] [Green Version] - Jones, A.E.; Puskarich, M.A.; Shapiro, N.I.; Guirgis, F.W.; Runyon, M.; Adams, J.Y.; Sherwin, R.; Arnold, R.; Roberts, B.W.; Kurz, M.C.; et al. Effect of levocarnitine vs placebo as an adjunctive treatment for septic shock: The Rapid Administration of Carnitine in Sepsis (RACE) randomized clinical TrialEffect of levocarnitine vs placebo as an adjunctive treatment for septic ShockEffect of levocarnitine vs placebo as an adjunctive treatment for septic shock. JAMA Netw. Open
**2018**, 1, e186076. [Google Scholar] [CrossRef] - Zhou, M.; Sharma, R.; Zhu, H.; Li, Z.; Li, J.; Wang, S.; Bisco, E.; Massey, J.; Pennington, A.; Sjoding, M.; et al. Rapid breath analysis for acute respiratory distress syndrome diagnostics using a portable two-dimensional gas chromatography device. Anal. Bioanal. Chem.
**2019**, 411, 6435–6447. [Google Scholar] [CrossRef] [PubMed] - Lemoine, N.P. Moving beyond noninformative priors: Why and how to choose weakly informative priors in Bayesian analyses. Oikos
**2019**, 128, 912–928. [Google Scholar] [CrossRef] [Green Version] - Armitage, E.G.; Godzien, J.; Alonso-Herranz, V.; López-Gonzálvez, Á.; Barbas, C. Missing value imputation strategies for metabolomics data. Electrophoresis
**2015**, 36, 3050–3060. [Google Scholar] [CrossRef] [PubMed] - Karpievitch, Y.V.; Dabney, A.R.; Smith, R.D. Normalization and missing value imputation for label-free LC-MS analysis. BMC Bioinform.
**2012**, 13 (Suppl. 16), S5. [Google Scholar] [CrossRef] [Green Version] - Lee, M.; Rahbar, M.H.; Brown, M.; Gensler, L.; Weisman, M.; Diekman, L.; Reveille, J.D. A multiple imputation method based on weighted quantile regression models for longitudinal censored biomarker data with missing values at early visits. BMC Med. Res. Methodol.
**2018**, 18, 8. [Google Scholar] [CrossRef] [Green Version] - Wei, R.; Wang, J.; Su, M.; Jia, E.; Chen, S.; Chen, T.; Ni, Y. Missing value imputation approach for mass spectrometry-based metabolomics data. Sci. Rep.
**2018**, 8, 663. [Google Scholar] [CrossRef] [Green Version] - Patti, G.J.; Yanes, O.; Siuzdak, G. Metabolomics: The apogee of the omics trilogy. Nat. Rev. Mol. Cell Biol.
**2012**, 13, 263–269. [Google Scholar] [CrossRef] - Carpenter, B.; Gelman, A.; Hoffman, M.D.; Lee, D.; Goodrich, B.; Betancourt, M.; Brubaker, M.; Guo, J.; Li, P.; Riddell, A. Stan: A probabilistic programming language. J. Stat. Softw.
**2017**, 76, 32. [Google Scholar] [CrossRef] [Green Version] - Gelman, A. Prior Choice Recommendations. Available online: https://github.com/stan-dev/stan/wiki/Prior-Choice-Recommendations (accessed on 22 April 2020).
- Goodrich, B.; Gabry, J.; Ali, I.; Brilleman, S. Rstanarm: Bayesian Applied Regression Modeling via Stan, R package version 2.19.3. 2020. Available online: https://mc-stan.org/rstanarm.

**Figure 1.**Simulation and model workflow. We began with a metabolomics dataset from patients with septic shock and prepared the data according to standard methods. We learned the mean and covariance of two outcome groups: survivors and non-survivors. Various parameters were adjusted to generate a simulated dataset corresponding to unique experimental conditions. We then ran a Bayesian logistic regression model and standard logistic regression +/− corrections for multiple testing to compare predicted results to the true results of the simulation.

**Figure 2.**A multilevel Bayesian approach offers increased power and controls for false discovery while providing a more accurate estimation of metabolite effect size relative to other statistical correction approaches. A multilevel Bayesian model (blue lines) and a standard logistic regression (labeled raw, purple lines) were fit on a simulated metabolic dataset where 40% of metabolites were defined to be significantly different between groups (survivors vs. non-survivors). Logistic regression models were further adjusted for multiple testing according to Bonferroni (green lines) and Benjamini–Hochberg (orange lines). Models were fit at different sample sizes per group without the presence of missing data as described in the methods. Model predictions are provided as: (

**A**) Power or True positive rate (TPR); (

**B**) False Discovery Rate (FDR); (

**C**) Average exaggeration ratio (AER) in estimated effect size. This is defined as the mean error over the set of metabolites that were significant and true (ST) for each model.

**Figure 3.**Two-stage ‘soft’ imputation methodology. We used Bayesian linear regression to impute missing metabolite observations, based on a censoring threshold and a user-defined number of correlated metabolites. The uncertainty in the imputed value, approximated by the standard deviation of the missing concentration, is accounted for upon fitting the subsequent multilevel Bayesian logistic regression.

**Figure 4.**A multilevel Bayesian approach can impute the value of missing metabolite observations and capture the uncertainty of the prediction, which improves the estimation of missing data relative to a standard metabolomics approach. Missing data were introduced into the simulated metabolomics dataset at a rate of 30% and imputation was completed using a multilevel Bayesian approach (

**A**) or a naïve approach (

**B**). In the Bayesian approach, the mean and standard deviation per missing metabolite concentration were computed using a left-censored Bayesian regression model for each metabolite based on the top eight most correlated metabolites and a pre-defined censoring threshold. The correlation of predicted and true metabolite concentration was 0.61 but improved to 0.65 when uncertainty in the prediction was accounted for using a weight function. In the naïve approach, the missing metabolite observation was calculated as the minimum concentration for that metabolite divided by two. The correlation of predicted vs. true metabolite concentrations using the naïve approach was 0.45.

**Figure 5.**In the presence of increasing missingness in metabolomics data, a multilevel Bayesian approach offers consistent model performance while providing a more accurate estimation of metabolite effect size relative to other statistical approaches. A multilevel Bayesian model (blue lines) and a standard logistic regression (labeled raw, purple lines) were fit on a simulated metabolic dataset where 40% of metabolites were defined to be significantly different between groups (survivors vs. non-survivors). Logistic regression models were further adjusted for multiple testing according to Bonferroni (green line) and Benjamini–Hochberg (orange line). Models were fit in the presence of increasing missing data as described in the methods. Model predictions are provided as: (

**A**) Power or True positive rate (TPR); (

**B**) False Discovery Rate (FDR); (

**C**) Average exaggeration ratio (AER) in estimated effect size. This is defined as the mean error over the set of metabolites that were significant and true (ST) for each model.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Gillies, C.E.; Jennaro, T.S.; Puskarich, M.A.; Sharma, R.; Ward, K.R.; Fan, X.; Jones, A.E.; Stringer, K.A.
A Multilevel Bayesian Approach to Improve Effect Size Estimation in Regression Modeling of Metabolomics Data Utilizing Imputation with Uncertainty. *Metabolites* **2020**, *10*, 319.
https://doi.org/10.3390/metabo10080319

**AMA Style**

Gillies CE, Jennaro TS, Puskarich MA, Sharma R, Ward KR, Fan X, Jones AE, Stringer KA.
A Multilevel Bayesian Approach to Improve Effect Size Estimation in Regression Modeling of Metabolomics Data Utilizing Imputation with Uncertainty. *Metabolites*. 2020; 10(8):319.
https://doi.org/10.3390/metabo10080319

**Chicago/Turabian Style**

Gillies, Christopher E., Theodore S. Jennaro, Michael A. Puskarich, Ruchi Sharma, Kevin R. Ward, Xudong Fan, Alan E. Jones, and Kathleen A. Stringer.
2020. "A Multilevel Bayesian Approach to Improve Effect Size Estimation in Regression Modeling of Metabolomics Data Utilizing Imputation with Uncertainty" *Metabolites* 10, no. 8: 319.
https://doi.org/10.3390/metabo10080319