^{1}

^{*}

^{2}

All rights reserved.

Epidemiological studies commonly test multiple null hypotheses. In some situations it may be appropriate to account for multiplicity using statistical methodology rather than simply interpreting results with greater caution as the number of comparisons increases. Given the one-to-one relationship that exists between confidence intervals and hypothesis tests, we derive a method based upon the Hochberg step-up procedure to obtain multiplicity corrected confidence intervals (CI) for odds ratios (OR) and by analogy for other relative effect estimates. In contrast to previously published methods that explicitly assume knowledge of P values, this method only requires that relative effect estimates and corresponding CI be known for each comparison to obtain multiplicity corrected CI.

Testing the statistical significance of multiple null hypotheses is a routine practice in epidemiologic and other types of biomedical research. By chance, the probability of wrongly rejecting one or more null hypotheses increases in proportion to the number of comparisons tested [

Various methods have been presented in the literature for controlling the type I error in the context of multiple hypothesis testing. The classic Bonferroni inequality [_{i}_{i}_{i}^{th} null hypothesis. In the simple case, α is apportioned evenly among the tests. Although the family-wise (FWER) and per family (PFER) error rates are preserved at the α level of significance, the Bonferroni procedure is known to be conservative, especially for highly correlated test statistics (i.e., type I error probability is less than the nominal level of α). For example, in the case of a study of multiple genetic polymorphisms, the assumption is that all variants being tested have equal probability of being truly associated with the outcome of interest and leads to overcorrection.[^{2} distribution, or are model independent [_{(1)}≤p_{(2)}≤…≤p_{(n)}, the Hochberg procedure rejects all hypothesis H_{i≤j}_{(j)}<α/(_{(j)} by (

Many researchers and journal editors increasingly recognize confidence intervals (CI) as the preferred measure for conveying statistical uncertainty of effect size estimates such as odds ratios (OR), relative risks (RR), and hazard ratios (HR), as P values have been commonly misunderstood and misinterpreted in the literature [

Below, we present a method to compute multiplicity corrected CI for OR and by analogy for other measures of relative risk, when no P values have been explicitly provided. This computationally simple method based upon the Hochberg step-up procedure only requires knowledge of individual test OR and CI, and the number of comparisons being tested.

The derivation of multiplicity corrected confidence intervals for a set of _{i}_{i}_{(1−α/2)} is the 100% × (1−α/2) percentile of a standard normal distribution, and solving for SE[log(OR_{i}

Substituting the right hand side of (3) into the equation for the 2-tailed

The corresponding P value is computed as:
_{i}_{(1)} ≤p_{(2)} ≤ ... ≤ p_{(n)} (with arbitrary ordering in the case of ties), the Hochberg multiplicity corrected P values denoted by “*” are computed as:
^{*}(^{*}[log (OR_{i}_{i}_{(i)} based upon the Hochberg step-up procedure can then be computed by substituting the above standard error from

By analogy, replacing OR in the above equations with other relative effect estimates such as RR or HR gives the corresponding multiplicity corrected CI for these measures. When P values are directly available for the individual hypothesis tests, the Hochberg multiplicity corrected CI may be computed directly beginning with

In this example, the conclusions regarding the association (or lack thereof) of (D) and the exposure do not substantively change after correction for multiplicity, thus lending weight to what otherwise might be only cautious interpretation referencing the possibility of a chance observation due to multiple comparisons. However, in other situations where CI is close to containing unity, a null hypothesis might no longer be rejected at least in strict statistical terms after correction for multiplicity.

Confidence intervals for OR, RR and other relative effect estimates are commonly reported in epidemiologic and public health literature without correction for multiple hypothesis testing. The failure to account for multiplicity may lead to inflation of type I error and over interpretation of any apparently “positive” findings. In the current paper, we show how CI for relative effect size estimates such as OR may be corrected for multiplicity by use of the Hochberg step-up procedure, a “closed-testing” method for protecting against making excessive false-positive inferences due to multiple comparisons.

Our method has several strengths. The corrected CI are simple to compute in standard statistical software packages that have function routines for determining percentiles and areas under a curve for a normal distribution. Since P values are not required for the original hypothesis tests, multiplicity corrected CI may be computed

Several limitations must be observed when applying our procedure for computing CI. The technique is not applicable when “exact sampling distribution” methods have been used to make statistical inferences. The Hochberg multiplicity correction also will inflate P values and related CI when one or more of the hypothesis tests involve a multi-level, logically related categorical variable (e.g., current smoker, former smoker, never smoker). In this case, it is unnecessary to correct CI for multiplicity for a logically related variable in multivariate space. The computed multiplicity corrected CI will be an approximate solution when the decimal accuracy is limited for the original OR and CI values. Accordingly, it is generally recommended that at least 2 or 3 significant digits of accuracy are available for published estimates when using this method in a ^{*}(^{*}(^{*}(_{(j)} and lesser ranked P values by

It also is important to note that correction for multiplicity may not be necessary or even desirable in some situations [

In the early days of the development of stepwise and closed tests for the control of type I error in multiple hypothesis testing, epidemiologists and statisticians commonly believed that joint CI could not be constructed for these procedures. However, it has been shown since that standard methods for constructing CI also readily apply to common stepwise multiplicity procedures.[

Although the most effective strategy to minimize type I error related to multiple comparisons is to simply reduce the number of comparisons, this in effect penalizes the researcher for conducting a more informative multivariable study [

confidence intervals

familywise error rate

hazard ratios

lower confidence interval

odds ratios

per family error rate

relative risks

standard error

Odds ratios (OR) and 95% confidence intervals (CI) for a hypothetical disease (D) and exposure to 3 dichotomously coded environmental risk factors, uncorrected and corrected for multiplicity

^{a} |
^{b} | ||||
---|---|---|---|---|---|

| |||||

^{*} |
^{*} | ||||

Non-Exposed | 587 / 2143 | 1.0 | Referent | 1.513 | Referent |

Exposed | 5 / 10 | 1.652 | [0.551–4.953] | [0.09–32] | |

Non-Exposed | 246 / 2143 | 1.0 | Referent | 1.068 | Referent |

Exposed | 1 / 10 | 1.151 | [0.142–9.324] | [0.14–9.3]^{c} | |

Non-Exposed | 141 / 2143 | 1.0 | Referent | 0.830 | Referent |

Exposed | 3 / 10 | 6.509 | [1.646–25.743] | [1.3–33] |

Adjusted for age and sex.

Using Hochberg step-up procedure.

Note: The multiplicity adjusted and unadjusted 95% CI will be equal in this case since the corresponding unadjusted P value for the Factor 2 comparison was the highest of the 3 comparisons and thus the multiplicative factor for p_{(j)} in

Multiplicity adjusted estimates.

This manuscript was made possible by a grant from NIH (1U01DK072493-01) entitled “Adolescent Bariatrics – Assessing Health Benefits and Risks.” Amy Lehman of the Biostatistics Center at The Ohio State University offered valuable comments during the writing of this manuscript and her knowledge and insight was greatly appreciated. The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of any funding agency.

_{2}random variables: a proof of the Simes conjecture