# A Method to Compute Multiplicity Corrected Confidence Intervals for Odds Ratios and Other Relative Effect Estimates

^{1}

^{2}

^{*}

## Abstract

**:**

## Introduction

_{i}denote the probability that hypothesis S

_{i}is incorrect, the Bonferroni probability for the joint null hypothesis may be written as:

_{i}denotes the P value corresponding to the i

^{th}null hypothesis. In the simple case, α is apportioned evenly among the tests. Although the family-wise (FWER) and per family (PFER) error rates are preserved at the α level of significance, the Bonferroni procedure is known to be conservative, especially for highly correlated test statistics (i.e., type I error probability is less than the nominal level of α). For example, in the case of a study of multiple genetic polymorphisms, the assumption is that all variants being tested have equal probability of being truly associated with the outcome of interest and leads to overcorrection.[3] The first order Bonferroni inequality may be improved upon given knowledge of the joint bivariate probabilities [2, 4, 5] or when the absolute value of the correlation coefficient is greater than 50% [2, 6]. However, these improvements have been limited in applied practice due to their restrictive nature. Several multiple testing procedures [7–9] based upon the “closure method” [10] and “Simes equality” [11] have been introduced and shown to be more powerful than the Bonferroni method for testing the intersection hypothesis [12–13]. Of the closure method based options, the Hochberg step-up multiple comparisons procedure [7] has gained popularity as being “easier to apply” than the more powerful procedures of Hommel [9] and Rom. The procedure also is uniformly more powerful than the Bonferroni-based, sequentially-rejective method of Holm [14] in many applied situations, e.g., when test statistics are uncorrelated, follow a multivariate normal or T

^{2}distribution, or are model independent [15–17]. Given an ordered set of P values, i.e., p

_{(1)}≤p

_{(2)}≤…≤p

_{(n)}, the Hochberg procedure rejects all hypothesis H

_{i≤j}if p

_{(j)}<α/(n−j+1) for any j=1, … , n. P values are incrementally corrected in order from smallest to largest by multiplying p

_{(j)}by (n−j+1), wherein the multiplicative factor for the largest P value is unity and thus remains the same after multiplicity correction.

## Methodology

_{i}(i=1 to n) in terms of the lower confidence interval (LCI) for OR

_{i}. Letting:

_{(1−α/2)}is the 100% × (1−α/2) percentile of a standard normal distribution, and solving for SE[log(OR

_{i})] we see that:

_{i}’s) from the lowest to highest values i.e., p

_{(1)}≤p

_{(2)}≤ ... ≤ p

_{(n)}(with arbitrary ordering in the case of ties), the Hochberg multiplicity corrected P values denoted by “*” are computed as:

^{*}(j) is bounded by unity.Rearranging (5) and solving for SE

^{*}[log (OR

_{i})] in the equation:

_{i}, i.e.:

_{(i)}based upon the Hochberg step-up procedure can then be computed by substituting the above standard error from eq. 9 into the following basic equation:

## Example

## Discussion

^{*}(j) (eq. 7) in rare cases may lead to an anomaly wherein p

^{*}(j) but not p

^{*}(j−1) will achieve statistical significance. In this situation, one might apply the de facto variation of multiplying p

_{(j)}and lesser ranked P values by j to obtain the corresponding Hochberg corrected P values.[28] And finally, the method should not be used if the logarithm of the effect estimate does not follow a normal distribution, or if the underlying observations are not independent and identically distributed.

## Conclusions

## Abbreviations:

CI | confidence intervals |

FWER | familywise error rate |

HR | hazard ratios |

LCI | lower confidence interval |

OR | odds ratios |

PFER | per family error rate |

RR | relative risks |

SE | standard error |

**Table 1:.**Odds ratios (OR) and 95% confidence intervals (CI) for a hypothetical disease (D) and exposure to 3 dichotomously coded environmental risk factors, uncorrected and corrected for multiplicity

Variable | Cases/Control | Odds Ratio^{a} | Uncorrected for Multiplicity | Corrected for Multiplicity^{b} | |
---|---|---|---|---|---|

95% CI (OR) | SE^{*} [log(OR)] | 95% CI^{*} (OR) | |||

Factor 1 | |||||

Non-Exposed | 587 / 2143 | 1.0 | Referent | 1.513 | Referent |

Exposed | 5 / 10 | 1.652 | [0.551–4.953] | [0.09–32] | |

Factor 2 | |||||

Non-Exposed | 246 / 2143 | 1.0 | Referent | 1.068 | Referent |

Exposed | 1 / 10 | 1.151 | [0.142–9.324] | [0.14–9.3]^{c} | |

Factor 3 | |||||

Non-Exposed | 141 / 2143 | 1.0 | Referent | 0.830 | Referent |

Exposed | 3 / 10 | 6.509 | [1.646–25.743] | [1.3–33] |

^{a}Adjusted for age and sex.

^{b}Using Hochberg step-up procedure.

^{c}Note: The multiplicity adjusted and unadjusted 95% CI will be equal in this case since the corresponding unadjusted P value for the Factor 2 comparison was the highest of the 3 comparisons and thus the multiplicative factor for p

_{(j)}in equation (7) will be equal to 1.

^{*}Multiplicity adjusted estimates.

## Acknowledgments

## References:

- Hsu, J. Multiple Comparisons: Theory and Methods; Boca Raton; Chapman & Hall/CRC, 1996. [Google Scholar]
- Hochberg, Y; Tamhane, A. Multiple Comparison Procedures; New York; Wiley, 1987. [Google Scholar]
- Campbell, H; Rudal, I. Interpretation of genetic association studies in complex disease. Pharmacogenomics J
**2002**, 2, 349–360. [Google Scholar] - Kounias, E. Bounds for the probability of a union of events with applications. Ann Math Statist
**1968**, 39, 2154–2158. [Google Scholar] - Stoline, M. The Hunter method of simultaneous inference and its recommended use for applications having large known correlation structures. JASA
**1983**, 78, 366–370. [Google Scholar] - Hunter, D. An upper bound for the probability of a union. J Appl Prob
**1976**, 13, 597–603. [Google Scholar] - Hochberg, Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika
**1988**, 75, 800–802. [Google Scholar] - Rom, D. A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika
**1990**, 77, 663–665. [Google Scholar] - Hommel, G. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika
**1988**, 75, 383–386. [Google Scholar] - Marcus, R; Peritz, E; Gabriel, K. On closed testing procedures with special reference to ordered analysis of variance. Biometrika
**1976**, 63, 655–660. [Google Scholar] - Simes, R. An improved Bonferroni procedure for multiple tests of significance. Biometrika
**1986**, 73, 751–754. [Google Scholar] - Denne, J; Koch, G. A sequential procedure for studies comparing multiple doses. Pharmaceut Statist
**2002**, 1, 107–118. [Google Scholar] - Liu, W. Multiple tests of a non-hierarchical finite family of hypotheses. J R Statist Soc B
**1996**, 58, 455–461. [Google Scholar] - Holm, S. A simple sequentially rejective multiple test procedure. Scand J Stat
**1979**, 6, 65–70. [Google Scholar] - Sarkar, S. Some probability inequalities for ordered MTP
_{2}random variables: a proof of the Simes conjecture. Ann Statist**1998**, 26, 494–504. [Google Scholar] - Sarkar, K; Chang, C. The Simes method for multiple hypothesis testing with positively dependent test statistics. JASA
**1997**, 92, 1601–1608. [Google Scholar] - Huang, Y; Hsu, J. Hochberg’s step-up method: cutting corners off Holm’s step-down method. Biometrika
**2007**, 94, 965–975. [Google Scholar] - Walker, A. Reporting the results of epidemiologic studies. Am J Public Health
**1986**, 76, 556–668. [Google Scholar] - Cooper, R; Wears, R; Schriger, D. Reporting research results: recommendations for improving communication. Ann Emerg Med
**2003**, 41, 561–564. [Google Scholar] - Connor, J. The value of a p-valueless paper. Am J Gastroenterol
**2004**, 99, 1638–1640. [Google Scholar] - Altman, D. Why we need confidence intervals. World J Surg
**2005**, 29, 554–556. [Google Scholar] - International Committee of Medical Journal Editors. Uniform requirements for manuscripts submitted to biomedical journals. Med Educ
**1999**, 33, 66–78. [Google Scholar] - Stefannsson, G; Kim, W; Hsu, J. On confidence sets in multiple comparisons. Gupta, SS, Berger, JO, Eds.; In Statistical Decision Theory and Related Topics IV; Springer-Verlag: New York, 1988; Volume 2, pp. 89–104. [Google Scholar]
- Hayter, A; Hsu, J. On the relationship between stepwise decision procedures and confidence sets. JASA
**1994**, 89, 128–136. [Google Scholar] - Holm, S. Multiple confidence sets based on stagewise tests. JASA
**1999**, 94, 489–495. [Google Scholar] - Ludbrook, J. Multiple inferences using confidence intervals. Clin Exp Pharmacol P
**2000**, 27, 212–215. [Google Scholar] - Shaffer, J. Multiple hypothesis testing. Annu Rev Psychol
**1995**, 46, 561–584. [Google Scholar] - Rochon, J. Update on repeated measurements; Short course notes - Roche Global Development: Nutley, NJ, 1996. [Google Scholar]
- Perry, J. Multiple-comparison procedures: A dissenting view. J Econ Entomol
**1986**, 79, 1149–1155. [Google Scholar] - Rothman, K. No adjustments are needed for multiple comparisons. Epidemiology
**1990**, 1, 43–46. [Google Scholar] - Perneger, T. What’s wrong with Bonferroni adjustments? BMJ
**1998**, 316, 1236–1238. [Google Scholar] - Berry, D; Hochberg, Y. Bayesian perspectives on multiple comparisons. J Statistical Plann Inference
**1999**, 82, 215–227. [Google Scholar] - Savitz, D; Olshan, A. Multiple comparisons and related issues in the interpretation of epidemiologic data. Am J Epidemiol
**1995**, 142, 904–908. [Google Scholar] - Bender, R; Lange, S. Multiple test procedures other than Bonferroni’s deserve wider use. BMJ
**1999**, 318, 600. [Google Scholar] - Aickin, M. Other method for adjustment of multiple testing exists. BMJ
**1999**, 318, 127. [Google Scholar]

© 2008 MDPI All rights reserved.

## Share and Cite

**MDPI and ACS Style**

Efird, J.T.; Nielsen, S.S.
A Method to Compute Multiplicity Corrected Confidence Intervals for Odds Ratios and Other Relative Effect Estimates. *Int. J. Environ. Res. Public Health* **2008**, *5*, 394-398.
https://doi.org/10.3390/ijerph5050394

**AMA Style**

Efird JT, Nielsen SS.
A Method to Compute Multiplicity Corrected Confidence Intervals for Odds Ratios and Other Relative Effect Estimates. *International Journal of Environmental Research and Public Health*. 2008; 5(5):394-398.
https://doi.org/10.3390/ijerph5050394

**Chicago/Turabian Style**

Efird, Jimmy Thomas, and Susan Searles Nielsen.
2008. "A Method to Compute Multiplicity Corrected Confidence Intervals for Odds Ratios and Other Relative Effect Estimates" *International Journal of Environmental Research and Public Health* 5, no. 5: 394-398.
https://doi.org/10.3390/ijerph5050394