A Note on the Robust Modification of the Ordered-Heterogeneity Test

Abstract

An ordered heterogeneity (OH) test is a test for a trend that combines a nondirectional heterogeneity test with the rank-order information specified under the alternative. A modified OH test introduced in 2006 can detect all possible patterns under the alternative with a relatively high power. Here, it is proposed to apply the modified OH test as a permutation test, which has the advantage that it requires only the exchangeability of the observations in the combined sample under the null hypothesis. No additional assumptions or simulations of critical values are necessary. A simulation study indicates that the permutation OH test controls the significance level even for small sample sizes and has a good power, comparable to competing tests. Moreover, the main advantage of the OH tests is their very broad applicability; they can always be applied when a heterogeneity test exists.

1. Introduction

Studies that compare more than two groups are widespread. In clinical dose-finding trials and other studies such as agricultural experiments, often increasing doses of the same drug or fertilizer are investigated. In these cases, there is an intrinsic order in the treatments. Usually only a few doses are studied, for instance, a placebo group and two or three doses. Then it is suitable and common to treat the dose level as a discrete variable [1]. Moreover, one might assume that an effect does not decline with increasing dose levels. Then, an appropriate alternative hypothesis is a trend: an increase in the effect with an increase in the dose level. In this case, the alternative is ordered, and a so-called trend test can be applied. Trend tests are particularly sensitive for detecting an ordered alternative.

For illustration, we consider a one-way design with k independent groups, appropriate, for example, in randomized parallel dose–response trials. The sample size of group i, i = 1, …, k, is denoted by $n_i$, and let $X_{i1},\dots ,X_{i{n}_i}$ be the i-th random sample distributed according to a distribution function $F_i$. Observations within samples are assumed to be independent and identically distributed, and mutual independence between samples is assumed. In the location-shift model, there is no difference under the null hypothesis

$${\mathrm{H}}_0:{\mu }_1={\mu }_2=\dots ={\mu }_k,$$

where ${\mu }_i$ is the location parameter associated with group i. We assume that the ${\mu }_i$ (i = 1, …, k) do not decline with increasing dose levels; therefore, trend tests are considered, and the null hypothesis is tested against the ordered alternative

$${\mathrm{H}}_1:{\mu }_1\le {\mu }_2\le \dots \le {\mu }_k\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}{\mu }_1<{\mu }_2.$$

Rice and Gaines [2,3] introduced the ordered-heterogeneity family of tests. The idea is to combine the p-value from a nondirectional heterogeneity test with the independent rank-order information specified under the alternative. The test statistic proposed by Rice and Gaines [2,3] is $r_s(1-P_H)$, where $P_H$ is the p-value of a heterogeneity test and $r_S$ is Spearman’s rank correlation. Note that Spearman’s rank correlation is Pearson’s product-moment correlation coefficient computed based on ranks [4]. Rice and Gaines [3] presented critical values for the ordered heterogeneity (OH) test determined under the assumptions that $P_H$ is uniform on (0, 1) and independent of $r_S$.

Rice and Gaines [2,3] pointed out that the OH test has a wide range of both parametric and nonparametric applications. They did not claim that the OH test has the highest power, but the advantage is its simplicity and broad applicability. The principle “permits one to easily convert almost any nondirectional heterogeneity test into a directional test” [2]. In the example discussed by Rice and Gaines [2], there are three populations of fruit flies with increasing variances under the alternative.

Neuhäuser and Hothorn [5] introduced modifications of the ordered heterogeneity test that are more robust. On the one hand, under the assumption ${\mu }_1\le {\mu }_2\le \dots \le {\mu }_k$, there are 2^k−1 different orderings. When we consider k = 3 as an example, there are the orderings ${\mu }_1<{\mu }_2<{\mu }_3$, ${\mu }_1={\mu }_2<{\mu }_3$, and ${\mu }_1<{\mu }_2={\mu }_3$ in addition to the null hypothesis ${\mu }_1={\mu }_2={\mu }_3$. The corresponding expected ranks (using mid-ranks for ties) of the non-null scenarios are (1, 2, 3), (1.5, 1.5, 3), and (1, 2.5, 2.5). For all the 2^k−1 − 1 scenarios under the trend alternative, Spearman’s rank correlation can be computed, and the maximum can be multiplied with $(1-P_H)$. Moreover, the observed ordering of the groups can be obtained not only using sample means but also using the mean ranks of the groups.

The usefulness of mean ranks was illustrated by Neuhäuser and Hothorn [5] with an example. The dry weight of spinach was investigated for five different doses. The smallest value was observed in the group with the second largest dose. Due to this outlier, the ranking of the sample means is (2, 4, 3, 1, 5). However, the ranking of the mean ranks is (1, 2, 4, 3, 5), which leads to a much larger correlation and test statistic.

Through the use of mean ranks and/or the maximum correlation, Neuhäuser and Hothorn [5] considered the following four different OH tests:

• OH1: ordering of the groups according to the sample means and test based on the expected ranks (1, 2, …, k);
• OH2: ordering of the groups according to the mean ranks and test based on the expected ranks (1, 2, …, k);
• OH3: ordering of the groups according to the sample means and test based on the maximum rank correlation out of the 2^k−1 − 1 possible expected ranks;
• OH4: ordering of the groups according to the mean ranks and test based on the maximum rank correlation out of the 2^k−1 − 1 possible expected ranks.

In the simulations presented by Neuhäuser and Hothorn [5], the OH4 test showed the best results with relatively high power for all possible patterns under the trend alternative, as well as for normal and non-normal distributions.

The critical values given by Rice and Gaines [3] cannot be utilized for the tests using the maximum rank correlation. Therefore, Neuhäuser and Hothorn [5] simulated critical values. To be precise, $P_H$ was generated from a uniform distribution on (0, 1), and correlations were obtained as rank correlations between independent samples of size k taken from simulated standard normal distributions. However, both the assumptions needed by Rice and Gaines [3] for their numerical calculation of critical values and the simulations performed by Neuhäuser and Hothorn [5] are no longer needed because, nowadays, permutation tests can easily be carried out with the test statistics $r_s(1-P_H)$ and $\max r_s(1-P_H)$, respectively. In a permutation test, inference is based on the permutation null distribution of the test statistic, which can be generated by considering all possible permutations under the null hypothesis; that is, the test statistic is computed for each permuted data set to obtain the null distribution. A p-value can be obtained from this null distribution as the proportion of permutations with an equal or more extreme value of the test statistic than the actually observed value of the test statistic [6]. In the considered one-way design, each permutation is a possible reallocation of the observed values to k groups of sizes $n_1,n_2,\dots ,n_k$. Often, a permutation test is preferable to traditional tests because of its exactness [7].

The number of possible permutations can be very large, especially for large values of k. However, in an approximate permutation test, the permutation distribution can be estimated based on a simple random sample of, e.g., B = 1000 permutations. According to Bonnini et al. [6], this number of 1000 permutations is sufficient; it gives a good degree of approximation and therefore reliable results. Marozzi [8] suggested 500 to 1000 permutations when estimating the size and power of a permutation test at level α = 0.05 in simulation studies. However, for smaller significance levels and for actual applications, more permutations should be used [8].

2. Materials and Methods

In this paper, we investigate (approximate) OH permutation tests based on $r_s(1-P_H)$ and ${\mathrm{m}\mathrm{a}\mathrm{x}r}_s(1-P_H)$ and compare these tests with two competing trend tests. On the one hand, we consider the test proposed by Jonckheere [9], which is a nonparametric trend test based on Mann–Whitney scores. This test usually is considered the standard non-parametric trend test for the one-way design [4]. On the other hand, we include the test of Shan et al. [10] that can be considered as an extension of Jonckheere’s test and additionally includes the rank difference between observations from different groups. There are further trend tests, as, e.g., a nonparametric test proposed by Cuzick [11], which is based on rank sums. This test was included in the simulation study presented by Shan et al. [10] and was considered not preferable to their proposed trend test [10].

A Monte Carlo simulation study was performed using R (version 4.4.1); 10,000 simulation runs were generated for each configuration, and approximate permutation tests were carried out based on B = 1000 permutations. The Kruskal–Wallis test was used as the heterogeneity test to obtain $P_H$. The trend tests of Jonckheere [9] and Shan et al. [10] were also performed as approximate permutation tests.

3. Results

3.1. Simulation Study

Here we present results for sample sizes $n_i$ = 10 (see

Table 1. Simulated size and power of different permutation trend tests for the location-shift model $X_{ij}=Z_{ij}+{\theta }_i$, where $Z_{ij}$ is distributed as specified (k = 3, $n_i$ = 10, α = 0.05).

Profile (${\mathit{\theta }}_1,{\mathit{\theta }}_2,{\mathit{\theta }}_3$)	Jonckheere	Shan et al.	OH1	OH2	OH3	OH4
Standard normal distribution
0, 0, 0	0.052	0.052	0.053	0.053	0.052	0.052
0, 0, 1.2	0.78	0.81	0.47	0.47	0.82	0.82
0, 0.3, 1.2	0.80	0.81	0.67	0.67	0.79	0.79
0, 0.6, 1.2	0.80	0.81	0.75	0.74	0.78	0.78
0, 0.9, 1.2	0.80	0.81	0.67	0.67	0.79	0.79
0, 1.2, 1.2	0.78	0.81	0.47	0.47	0.82	0.82
Exponential distribution with mean 1
0, 0, 0	0.046	0.046	0.049	0.048	0.049	0.047
0, 0, 1.2	0.91	0.94	0.49	0.50	0.94	0.94
0, 0.3, 1.2	0.94	0.95	0.73	0.79	0.92	0.93
0, 0.6, 1.2	0.94	0.94	0.82	0.89	0.91	0.92
0, 0.9, 1.2	0.93	0.93	0.72	0.81	0.89	0.91
0, 1.2, 1.2	0.88	0.91	0.48	0.48	0.89	0.89
t distribution with df = 3
0, 0, 0	0.044	0.046	0.047	0.045	0.047	0.046
0, 0, 1.2	0.61	0.64	0.41	0.42	0.63	0.63
0, 0.3, 1.2	0.63	0.64	0.49	0.54	0.59	0.61
0, 0.6, 1.2	0.64	0.64	0.53	0.58	0.57	0.60
0, 0.9, 1.2	0.63	0.64	0.49	0.54	0.59	0.61
0, 1.2, 1.2	0.61	0.64	0.40	0.41	0.63	0.64

The test with the largest power is highlighted in bold for each scenario.

,

Table 2. Simulated size and power of different permutation trend tests for the location-shift model $X_{ij}=Z_{ij}+{\theta }_i$, where $Z_{ij}$ is distributed as specified (k = 4, $n_i$ = 10, α = 0.05).

Profile (${\mathit{\theta }}_1,{\mathit{\theta }}_2,{\mathit{\theta }}_3,{\mathit{\theta }}_4$)	Jonckheere	Shan et al.	OH1	OH2	OH3	OH4
Standard normal distribution
0, 0, 0, 0	0.050	0.051	0.048	0.048	0.051	0.050
0, 0, 0, 1.2	0.73	0.77	0.47	0.47	0.74	0.74
0, 0, 0.8, 1.2	0.90	0.90	0.85	0.85	0.90	0.90
0, 0, 1.2, 1.2	0.93	0.94	0.82	0.82	0.96	0.96
0, 0.4, 0.8, 1.2	0.85	0.85	0.82	0.81	0.81	0.81
0, 1.2, 1.2, 1.2	0.73	0.78	0.47	0.47	0.74	0.74
Exponential distribution with mean 1
0, 0, 0, 0	0.052	0.053	0.052	0.052	0.051	0.052
0, 0, 0, 1.2	0.87	0.92	0.49	0.50	0.89	0.89
0, 0, 0.8, 1.2	0.98	0.98	0.91	0.95	0.97	0.98
0, 0, 1.2, 1.2	0.98	0.99	0.86	0.87	0.98	0.99
0, 0.4, 0.8, 1.2	0.97	0.97	0.89	0.95	0.93	0.95
0, 1.2, 1.2, 1.2	0.83	0.88	0.49	0.49	0.83	0.83
t distribution with df = 3
0, 0, 0, 0	0.048	0.049	0.047	0.048	0.046	0.048
0, 0, 0, 1.2	0.56	0.59	0.39	0.41	0.54	0.56
0, 0, 0.8, 1.2	0.75	0.75	0.64	0.70	0.71	0.74
0, 0, 1.2, 1.2	0.79	0.81	0.68	0.72	0.80	0.83
0, 0.4, 0.8, 1.2	0.68	0.68	0.58	0.64	0.59	0.63
0, 1.2, 1.2, 1.2	0.56	0.60	0.39	0.40	0.54	0.56

The test with the largest power is highlighted in bold for each scenario.

and

Table 3. Simulated size and power of different permutation trend tests for the location-shift model $X_{ij}=Z_{ij}+{\theta }_i$, where $Z_{ij}$ is distributed as specified (k = 5, $n_i$ = 10, α = 0.05).

Profile (${\mathit{\theta }}_1,{\mathit{\theta }}_2,{\mathit{\theta }}_3,{\mathit{\theta }}_4,{\mathit{\theta }}_5$)	Jonckheere	Shan et al.	OH1	OH2	OH3	OH4
Standard normal distribution
0, 0, 0, 0, 0	0.047	0.049	0.049	0.047	0.049	0.049
0, 0, 0, 0, 1.2	0.68	0.72	0.44	0.44	0.65	0.65
0, 0, 0, 1.2, 1.2	0.95	0.96	0.90	0.90	0.97	0.97
0, 0, 1.2, 1.2, 1.2	0.95	0.96	0.91	0.91	0.97	0.97
0, 1.2, 1.2, 1.2, 1.2	0.68	0.72	0.43	0.43	0.65	0.65
0, 0, 0.6, 1.2, 1.2	0.95	0.96	0.95	0.95	0.96	0.96
0, 0.3, 0.6. 0.9, 1.2	0.88	0.89	0.86	0.85	0.85	0.85
Exponential distribution with mean 1
0, 0, 0, 0, 0	0.051	0.052	0.051	0.049	0.053	0.050
0, 0, 0, 0, 1.2	0.83	0.88	0.48	0.48	0.80	0.80
0, 0, 0, 1.2, 1.2	0.99	0.995	0.93	0.94	0.99	0.997
0, 0, 1.2, 1.2, 1.2	0.99	0.99	0.92	0.93	0.99	0.99
0, 1.2, 1.2, 1.2, 1.2	0.79	0.84	0.46	0.46	0.75	0.74
0, 0, 0.6, 1.2, 1.2	0.99	0.995	0.98	0.99	0.99	0.99
0, 0.3, 0.6, 0.9, 1.2	0.99	0.98	0.93	0.97	0.94	0.97
t distribution with df = 3
0, 0, 0, 0, 0	0.046	0.046	0.049	0.048	0.049	0.049
0, 0, 0, 0, 1.2	0.53	0.55	0.37	0.37	0.49	0.50
0, 0, 0, 1.2, 1.2	0.87	0.86	0.76	0.81	0.82	0.87
0, 0, 1.2, 1.2, 1.2	0.84	0.86	0.76	0.82	0.82	0.87
0, 1.2, 1.2, 1.2, 1.2	0.53	0.56	0.37	0.38	0.49	0.50
0, 0, 0.6, 1.2, 1.2	0.86	0.86	0.77	0.84	0.80	0.84
0, 0.3, 0.6, 0.9, 1.2	0.74	0.74	0.64	0.70	0.64	0.68

The test with the largest power is highlighted in bold for each scenario.

) and $n_i$ = 20 (see Figure 1). Even for the small sample size $n_i$ = 10, the actual type I error rate (size) is close to the nominal significance level. Although permutation tests guarantee the significance level α, there are simulated actual type I error rates slightly larger than the chosen α of 5% due to the error margin of the simulation. However, the approximate OH permutation trend tests, as well as the competing tests, have good and similar type I error rates.

In Table 1, Table 2 and Table 3, the test with the largest power is highlighted in bold for each scenario. We can confirm that OH4 is the best of the four OH permutation trend tests. This test has a good power, which is comparable to that of the standard trend tests of Jonckheere or Shan et al. Although in several considered scenarios the test proposed by Shan et al. [10] has the highest power, there are also scenarios where OH4 has the highest power (see, e.g., Figure 1). As expected, because no single trend test is uniformly most powerful [3], the simulation study does not show a clear winner. The tests OH4 and Jonckheere do not differ much.

3.2. Example

Raw data of reaction times (in seconds) of mice in a control group and three further groups with increasing doses were presented by Shirley [12]. The sample size is ten per group. Since the distribution of the reaction time is highly skewed [12], non-parametric methods seem preferable. With the data from all four groups, trend tests give a clear significance with, e.g., p < 0.0001 in the Jonckheere test. Therefore, hereafter, the control group is omitted for illustrative purposes. With the remaining three groups, permutation tests were performed with B = 100,000 permutations.

Although the Kruskal–Wallis test is not significant with a p-value larger than 10%, the OH tests that combine the Kruskal–Wallis test with Spearman’s rank correlation give the p-values 0.0198 (OH1), 0.0196 (OH2), 0.0200 (OH3), and 0.0198 (OH4). There are only marginal differences between the four different OH tests in this example, where the means give the same ordering (1, 2, 3) as the mean ranks. The other trend tests result in the p-values 0.0207 for the Jonckheere test and 0.0224 for the test proposed by Shan et al. [10]. Thus, all trend tests reject the null hypothesis at α = 0.05, but the OH tests’ p-values are slightly smaller.

4. Discussion and Conclusions

The robust modification of the ordered heterogeneity test based on the maximum correlation has a relatively high power for all possible patterns under the alternative; furthermore, ordering the groups according to mean ranks is useful, especially for non-normal distributions [5]. The test based on $\max r_s(1-P_H)$ can be performed as a permutation test, which only requires the exchangeability of the observations in the combined sample under the null hypothesis. Exchangeability is fulfilled for random variables if any permutation has the same joint distribution function, or in other words, when the labels on the observations can be exchanged, under the null hypothesis of no difference, without affecting the results [13]. Thus, permutation tests are possible if there is random assignment of treatments to experimental units [6], a situation common in clinical trials [14]. Therefore, trend tests based on permutations have important applications in clinical trials. Beyond exchangeability, no further assumptions are needed for the computation or simulation of critical values [3,5].

Here, it is shown that the OH permutation tests control the type I error rate and have a good power comparable to standard trend tests such as the Jonckheere or Shan et al. tests. Although our simulation is limited to location-shift models, the important advantage of the OH tests is their very broad applicability. There are trend tests for some more complex designs, such as the test proposed by Page [15] for a two-way design or by Bonnini et al. [16] for multivariate binary data based on combined permutation tests. In many other complex designs, a heterogeneity test exists while a trend test might not exist or is not implemented in statistical software. Then, the heterogeneity test can be used in order to create an OH trend test. Standard tests are sufficient for the one-way design, but OH tests are extremely useful for more complex designs. Moreover, even in the one-way design, an OH trend test is reasonable for other test problems than the comparison of locations. For example, Bisschop et al. [17] performed an OH test based on Levene’s test of equality of variances.

Note that permutation tests are available for complex designs. A possible way is to permute the residuals [18]; Potter [19] proposed this approach also for logistic regression. Moreover, permutation approaches for repeated measurements, factorial and some other complex designs are presented by Pesarin [20], Pesarin and Salmaso [21], as well as Brunner et al. [22]. Basso et al. [23] discuss permutation tests for two-way designs and unreplicated factorial designs. Resampling-based methods such as permutation tests are discussed by Westfall and Young [24] for multiple test problems.

In our study, four different OH tests were investigated. According to the simulation results, we suggest OH4 for practical applications, in particular when distributional assumptions are uncertain. As a permutation test, this test can also be applied in case of small sample sizes and for larger sample sizes based on a simple random sample of permutations. The OH4 test uses mean ranks and is based on the maximum rank correlation.

In situations where exchangeability is not fulfilled, as for instance in the Behrens–Fisher problem when testing for differences in location under heteroscedasticity, bootstrap resampling might be an alternative. In future work, we will investigate OH tests based on the modified Brown–Forsythe test according to Mehrotra [25]. This heterogeneity test was proposed as an alternative to the ANOVA F test when data might be non-normal with heterogeneous variances [26]. In this situation, the OH test could be carried out based on a separate-sample (group-wise) bootstrap.