Combined Permutation Tests for Pairwise Comparison of Scale Parameters Using Deviances

Richter, Scott J.; McCann, Melinda H.

doi:10.3390/stats7020021

Open AccessArticle

Combined Permutation Tests for Pairwise Comparison of Scale Parameters Using Deviances

by

Scott J. Richter

^1,* and

Melinda H. McCann

²

¹

Department of Mathematics and Statistics, University of North Carolina at Greensboro, Greensboro, NC 27402, USA

²

Department of Statistics, Oklahoma State University, Stillwater, OK 74075, USA

^*

Author to whom correspondence should be addressed.

Stats 2024, 7(2), 350-360; https://doi.org/10.3390/stats7020021

Submission received: 27 February 2024 / Revised: 21 March 2024 / Accepted: 25 March 2024 / Published: 28 March 2024

(This article belongs to the Section Statistical Methods)

Download

Browse Figure

Review Reports Versions Notes

Abstract

:

Nonparametric combinations of permutation tests for pairwise comparison of scale parameters, based on deviances, are examined. Permutation tests for comparing two or more groups based on the ratio of deviances have been investigated, and a procedure based on Higgins’ RMD statistic was found to perform well, but two other tests were sometimes more powerful. Thus, combinations of these tests are investigated. A simulation study shows a combined test can be more powerful than any single test.

Keywords:

deviance; scale parameter; pairwise comparison; permutation test; combined test

1. Introduction

Tests for homogeneity of scale are of interest in many areas of application, including industrial quality assurance, agricultural production and education [1]. Parametric tests for comparing scale (e.g., [2,3,4]) are generally not robust to nonnormality (see [5]). Consequently, more robust alternatives are of interest.

An approximate test using the ANOVA F-test on the absolute deviations from the mean was proposed [6]. Using absolute deviations from the median (referred to as deviances in the remainder of this paper), referred to as the W50 test, was later suggested [7]. However, no uniformly best test for scale has been demonstrated in the literature. In fact, without more stringent distributional assumptions, the minimal sufficient statistic would generally be the n-dimensional vector of order statistics. Thus, no single statistic exists that summarizes the information contained in the data, and a uniformly best test statistic does not generally exist. In spite of this, the W50 test has been recommended as a computationally simple test showing good overall performance with respect to power and robustness to nonnormality in several comparative studies ([8,9,10]). More recently, a study [5] compared 25 omnibus tests for homogeneity of variance and recommended the W50 test as “superior”. A modification of Levene’s test [6] (referred to as OB) was proposed [11] which has been recommended over the W50 test for light-tailed distributions [12]. The W50 and OB tests, as well as permutation versions of these tests, were evaluated [5] and it was found that the permutation versions tended to be more robust and have higher power. The W50 test was recommended as a computationally simple robust test, as was the permutation version of the OB test for symmetric and lighter-tailed skewed distributions. Another test for scale utilizing deviances, based on the ratio of the mean deviances, was also proposed [13]. This test will be referred to as the RMD test. The RMD test was found [14] to be generally superior to W50 and OB, although there were still cases where each of W50 and OB had higher power. Since no test has been found to be uniformly superior, it is of interest to develop a test that combines these three tests. A combined test of scale parameters based on the IQR was studied [15] and the combined test was found to be more powerful than its constituent tests in some scenarios. Similarly, we will investigate nonparametric combinations of the RMD, W50 and OB tests to determine if combining the tests can provide increased power compared to individual tests.

2. Methods for Comparing Scale Parameters

Consider a one-way layout with t treatments and

n_{i}

observations per treatment. We assume a location-scale model,

y_{i j} = μ_{i} + σ_{i} ε_{i j}

,

i = 1, \dots, t, j = 1, \dots, n_{i}

, where

μ_{i}

and

σ_{i}

are the location and scale parameters, respectively, of treatment i, and

ε_{i j}

are independent and identically distributed with median 0. It is desired to test

H_{0} : σ_{1} = σ_{2} = \dots = σ_{t}

versus

H_{a} : σ_{i} \neq σ_{j}

for some i and j.

2.1. Brown–Forsythe (W50) Test

First, compute the deviances,

{\tilde{z}}_{i j} = |y_{i j} - {\tilde{y}}_{i}|

, where

\tilde{y}

is the sample median. The ANOVA F test is performed on these scores, and the p-value is based on the F distribution with

t - 1

and

n - t

degrees of freedom [7].

2.2. Higgins’ (RMD) Test

The statistic is defined as,

R M D = \frac{\max ({\bar{\tilde{z}}}_{i}, {\bar{\tilde{z}}}_{j})}{\min ({\bar{\tilde{z}}}_{i}, {\bar{\tilde{z}}}_{j})}

, where

{\bar{\tilde{z}}}_{i}

is the mean of the deviances,

{\tilde{z}}_{i j}

, for treatment i. The deviances

{\tilde{z}}_{i j} = |y_{i j} - {\tilde{y}}_{i}|

are the same as those used by the W50 test. The permutation distribution of the RMD statistic was used to calculate a p-value [13].

2.3. O’Brien’s (OB) Test

First, compute the scores

r_{i j} (w) = \frac{[(w + n_{i} - 2) n_{i j} {(y_{i j} - {\bar{y}}_{i})}^{2} - w s_{i}^{2} (n_{i} - 1)]}{[(n_{i} - 1) (n_{i} - 2)]}

, where

0 \leq w \leq 1

. At one extreme, when

w = 0

, the statistic reduces to

r_{i j} (0) = \frac{n_{i} {(y_{i j} - {\bar{y}}_{i})}^{2}}{n_{i} - 1}

, which is a slight modification of Levene’s test, which uses

{\bar{z}}_{i j}^{2} = {(y_{i j} - {\bar{y}}_{i})}^{2}

. At the other extreme, when

w = 1

,

r_{i j} (1) = q_{i j} = \frac{[n_{i} {(y_{i j} - {\bar{y}}_{i})}^{2} - s_{i}^{2}]}{n_{i} - 2} = n_{i} s_{i}^{2} - (n_{i} - 1) s_{i - 1}^{2},

which was referred to as a “jackknife pseudovalue of

s_{i}^{2}

[11]”. The ANOVA F test is performed on these scores and, the p-value is based on the F distribution with

t - 1

and

n - t

degrees of freedom. Tests based on

{\bar{z}}_{i j}^{2}

have been shown to have inflated Type I error rates, while those based on

q_{i j}

tend to have low power. Since

r (w)

is a weighted average of the two tests, it provides a way to balance the drawbacks of the two tests. A “utility” value of

w = 0.5

was suggested for most situations [11], and this is the value employed in this study.

2.4. Permutation Tests

While the permutation test using the RMD statistic was suggested [13], the W50 and OB tests described previously were proposed as approximate tests based on the F distribution. However, p-values for W50 and OB can also be calculated using permutation distributions. A simulation study [1] found for the two-treatment case that the permutation versions tended to be more robust and have greater power than the approximate tests. Thus, we will consider only the permutation versions of these combined tests. Test statistics will be computed for a large number of random reassignments of observations to treatments, and the p-value will be calculated as the proportion of values of the permutation distribution that is at least as extreme as the observed test statistic value.

3. Combined Tests

A two-step approach to create a nonparametric combination of dependent tests was proposed [16] and described as follows:

Step 1. Analyze the data using the tests of interest, referred to as partial tests;
Step 2. Combine the partial tests to assess the global hypothesis.

Several different combining functions have been developed that satisfy the properties required for a suitable combining function [16]. Since the relative power of different combining functions can vary across conditions, we consider combined tests using three of the best-known combining functions: the Fisher, Liptak and Tippett combining functions [15].

Let

λ_{i}

be the p-value associated with the ith test to be combined. Then, the test statistics for the Fisher, Liptak and Tippett functions are

The Fisher combining function is $T_{F} = - \sum_{i} \ln (λ_{i})$ ;
The Liptak combining function is $T_{L} = \sum_{i} Φ^{- 1} (1 - λ_{i})$ ;
The Tippett combining function is $T_{T} = \max_{i} (1 - λ_{i})$ .

The Tippett function tends to have the highest power when one or a few, but not all, of the constituent tests reject the null hypothesis; the Liptak function tends to have the highest power when all tests reject the null hypothesis; the power of the Fisher function will tend to lie between the other two, making it the more general option and thus probably the most popular [16]. The combined tests are carried out as follows [16].

Compute the observed test statistic value ( $T_{F}$ , $T_{L}$ , $T_{T}$ ) according to the above definitions, using the permutation p-values of RMD, W50 and OB.
To compute the permutation test p-value associated with each combined statistic:
- For the ith statistic in the permutation distributions constructed for RMD, W50 and OB, compute the ith partial p-value as the proportion of test statistic values at least as large as the ith statistic value.
- Using the partial p-values for RMD, W50 and OB, use the respective combining function to compute a test statistic value ( $T_{F}$ , $T_{L}$ , $T_{T}$ ) for each permutation. This results in a permutation distribution for each of the combined statistics.
- For each combined test, the permutation p-value is then the proportion of values in the permutation distribution at least as large as the observed statistic value.

Note that all tests are based on the same set of randomly generated permutations.

Since the RMD, W50 and OB tests were each most powerful for at least some scenarios in past simulations (e.g., [5]), combinations of these three tests will be examined. In addition, since RMD and W50 were usually more powerful than OB, a combination of only RMD and W50 will also be considered. The p-values for each of the constituent tests in each combination will be estimated using the permutation distribution of the statistic. The powers Type I error rates of the Fisher, Liptak and Tippett combining functions will be estimated and compared, and these will also be compared to those of the individual tests.

4. Strong Familywise Error Rate Control for Pairwise Comparisons

The familywise error rate (FWER) will be controlled using the technique of Richter and McCann [17]. Richter and McCann [17] proposed a restricted permutation method to provide strong control of the familywise error rate (FWER) for pairwise comparison of location parameters. This method will be extended to the present case of comparing scale parameters as follows. First, the two-sample test statistic for a given method will be calculated for each of the possible

t (t - 1) / 2

pairs of treatments. Then, the maximum value of the test statistic across all pairs will be calculated. Next, observations will be reassigned at random to treatments within each pair of treatments, a test statistic calculated for each pair of treatments, and the maximum value determined. This will be repeated many times to build the permutation distribution, and the p-value for comparing each pair of treatments will be calculated as the proportion of values in the permutation distribution that is at least as extreme as the observed value.

5. Simulation Study

5.1. Procedures Studied

A simulation study estimated and compared the familywise Type I error rate and “any-pair” power (probability of detecting at least one true difference) of the methods described in Section 2:

RMD: Higgins RMD procedure.
W50: Brown and Forsyth’s W50 test.
OB: O’Brien’s method using means.
$F_{3} :$ Fisher’s combination test of RMD, W50 and OB.
$F_{2} :$ Fisher’s combination test of RMD and W50.
$L_{3} :$ Liptak’s combination test of RMD, W50 and OB.
$L_{2} :$ Liptak’s combination test of RMD and W50.
$T_{3} :$ Tippett’s combination test of RMD, W50 and OB.
$T_{2} :$ Tippett’s combination test of RMD and W50.

5.2. Sample Sizes and Differences in Scale Parameters

Both equal and unequal sample size settings were examined for five treatments. For equal sample size cases,

n_{i} = 10

and

n_{i} = 30

were used. Unequal sample size cases, settings of

n_{1} = 5, n_{2} = 5, n_{3} = 10, n_{4} = 15, n_{5} = 15

and

n_{1} = 10, n_{2} = 10, n_{3} = 20, n_{4} = 30, n_{5} = 30

were utilized. Maximum scale parameter ratios,

\frac{σ_{\{m a x\}}}{σ_{\{m i n\}}}

, ranging from 1 to 5 were examined, with different patterns of smaller ratios present. Settings of

(σ, 1,1, 1,1)

and

(σ, (σ + 1) / 2,1, 1,1)

were used. The first setting we refer to as the “single extreme scale parameter” setting, while the second setting has an intermediate scale value midway between the minimum (1) and maximum (σ). The specific settings used for

{(σ}_{1}, σ_{2}, σ_{3}, σ_{4}, σ_{5}

) were as follows:

1. $(1,1, 1,1, 1)$ 2. $(3,1, 1,1, 1)$ 3. $(3,2, 1,1, 1)$ 4. $(5,1, 1,1, 1)$ 5. $(5,3, 1,1, 1)$ .

5.3. Distributions

Several different g and h distributions [18] were used to simulate data from distributions with different characteristics. g and h distributions are monotonic functions of normal distributions and allow investigation of nonnormal distributions with specific characteristics. The g-and-h random variable is defined as

Y_{g, h} (Z) = (\frac{\exp (g Z) - 1}{g}) \exp (\frac{h Z^{2}}{2}),

where

Z \sim N (0,1) .

When g = h = 0,

Y_{g, h} (Z) \sim N (0,1) .

Nonzero values of g increase the skewness and positive values of h increase the elongation (tail heaviness) of the distribution. Changing the values of g and h does not affect the location of the distribution. The following cases were considered, and representative plots shown in Figure 1:

g = 0, h = 0—Normally distributed (symmetric, light tails);
g = 0, h = 0.4—Symmetric, moderately heavy tails;
g = 0, h = 0.8—Symmetric, very heavy tails;
g = 0.4, h = 0—Moderately skewed, light tails;
g = 0.8, h = 0—Heavily skewed, light tails;
g = 0.4, h = 0.4—Moderately skewed, moderately heavy tails;
g = 0.8, h = 0.4—Heavily skewed, moderately heavy tails.

Type I error rate and power were estimated based on 1000 randomly selected data sets from each distribution, for each setting of sample sizes and scale parameter patterns. It has been suggested [19] that only 253 random permutations are necessary with 1000 random data sets if the goal of the simulation is to estimate the power of a test and only a “rough” estimate of the permutation p-value is required, while a random sample of at least 1600 permutations was recommended [20] to estimate the exact p-value for a permutation test. Since precise estimation of the permutation test p-values was considered important, a conservative sample of 1999 random permutations was utilized, and thus the permutation distribution for each test was based on 2000 values: the observed test statistic value plus 1999 values based on random permutations of the observed data.

6. Simulation Results

6.1. Familywise Type I Error

All tests were robust in the sense that estimated rates of Type I error were close to the nominal level of 0.05 (See Table 1, Table 2, Table 3, Table 4, Table 5 and Table 6) with only one exceeding 0.075 (0.084 for RMD in the equal sample

n_{i} = 30

case, g = 0.8, h = 0.4). Note that in the tables, the first row of each distribution represents the equal scale case, and thus the value given is the estimated Type I error rate.

6.2. Any-Pair Power

When sample sizes were equal (Table 1 and Table 2), RMD tended to have the highest power, although in some cases the Fisher or Liptak combined test was most powerful.

When sample sizes were small and unequal and the larger scales were associated with the smaller samples (Table 3), the F₂ and L₂ combined tests were most powerful for all scale configurations, with L₂ usually having the higher power. The lone exception was when the distribution was symmetric with very heavy tails (g = 0, h = 0.8) where the RMD had similar power to F₂ and L₂. When the sample sizes increased to

n_{i} = 10, 10, 20, 30, 30

, however, the power advantage of the combined tests over RMD tended to diminish, except for the skewed, light-tailed distributions, where the combined tests were still more powerful (See Table 4).

Neither of the Tippett combined tests was as powerful as the Liptak and Fisher versions.

When the sample sizes were small and unequal but the larger scales were associated with the larger samples (Table 5 and Table 6), L₂ and F₂ had the highest power for normal and moderately skewed-only distributions. Meanwhile, RMD had the highest power for all distributions with heavy tails (

h = 0.4, 0.8)

. As before, as sample sizes increased, the power advantages of the combined tests diminished while RMD maintained power advantages for heavier-tailed distributions.

Table 1. Proportion of at least one rejection at

α = 0.05

, five treatments, equal samples of size

n_{i} = 10

.

Table 1. Proportion of at least one rejection at

α = 0.05

, five treatments, equal samples of size

n_{i} = 10

.

Distribution	Scale	Method
Distribution	${(σ}_{1} σ_{2} σ_{3} σ_{4} σ_{5})$	W50	OB	RMD	F₃	F₂	L₃	L₂	T₃	T₂
g = 0, h = 0	11111	0.040	0.016	0.039	0.039	0.042	0.041	0.043	0.025	0.034
	31111	0.669	0.620	0.742	0.708	0.726	0.708	0.727	0.689	0.710
	32111	0.609	0.466	0.689	0.648	0.692	0.655	0.697	0.604	0.632
	51111	0.911	0.850	0.965	0.935	0.954	0.933	0.950	0.944	0.957
	53111	0.889	0.697	0.970	0.925	0.956	0.924	0.957	0.929	0.944
g = 0, h = 0.4	11111	0.010	0.000	0.065	0.020	0.034	0.014	0.031	0.030	0.043
	31111	0.099	0.028	0.244	0.155	0.205	0.126	0.204	0.191	0.202
	32111	0.084	0.016	0.274	0.141	0.212	0.112	0.205	0.197	0.225
	51111	0.285	0.090	0.476	0.378	0.445	0.331	0.446	0.411	0.433
	53111	0.192	0.046	0.495	0.332	0.434	0.247	0.421	0.409	0.433
g = 0, h = 0.8	11111	0.004	0.000	0.071	0.014	0.031	0.005	0.024	0.036	0.051
	31111	0.017	0.003	0.140	0.057	0.098	0.032	0.084	0.092	0.110
	32111	0.011	0.001	0.182	0.060	0.110	0.034	0.088	0.108	0.133
	51111	0.053	0.011	0.237	0.130	0.192	0.079	0.174	0.189	0.208
	53111	0.034	0.007	0.286	0.129	0.208	0.070	0.182	0.214	0.240
g = 0.4, h = 0	11111	0.042	0.020	0.049	0.037	0.053	0.036	0.056	0.031	0.037
	31111	0.620	0.552	0.656	0.660	0.681	0.665	0.686	0.627	0.630
	32111	0.549	0.402	0.637	0.597	0.630	0.599	0.638	0.553	0.585
	51111	0.891	0.816	0.942	0.933	0.943	0.928	0.942	0.931	0.935
	53111	0.842	0.614	0.931	0.900	0.928	0.889	0.928	0.889	0.906
g = 0.4, h = 0.4	11111	0.007	0.001	0.062	0.015	0.029	0.011	0.028	0.023	0..030
	31111	0.090	0.028	0.240	0.146	0.204	0.118	0.200	0.184	0.200
	32111	0.081	0.014	0.272	0.130	0.216	0.099	0.201	0.195	0.222
	51111	0.251	0.090	0.442	0.343	0.418	0.301	0.415	0.384	0.406
	53111	0.182	0.040	0.477	0.310	0.406	0.248	0.390	0.374	0.406
g = 0.8, h = 0	11111	0.034	0.014	0.058	0.042	0.047	0.036	0.052	0.033	0.042
	31111	0.426	0.359	0.472	0.514	0.490	0.513	0.497	0.465	0.458
	32111	0.346	0.236	0.468	0.432	0.451	0.430	0.459	0.413	0.413
	51111	0.765	0.658	0.800	0.835	0.826	0.826	0.828	0.813	0.802
	53111	0.640	0.426	0.801	0.780	0.791	0.754	0.799	0.755	0.762
g = 0.8, h = 0.4	11111	0.011	0.002	0.063	0.015	0.030	0.009	0.027	0.028	0.038
	31111	0.074	0.025	0.195	0.138	0.168	0.104	0.164	0.154	0.171
	32111	0.064	0.014	0.231	0.131	0.179	0.089	0.167	0.156	0.185
	51111	0.186	0.065	0.384	0.294	0.348	0.242	0.346	0.313	0.338
	53111	0.138	0.032	0.430	0.272	0.356	0.204	0.346	0.333	0.364

Table 2. Proportion of at least one rejection at

α = 0.05

, five treatments, equal samples of size

n_{i} = 30

. Cases that were uninformative for comparing methods were omitted.

Table 2. Proportion of at least one rejection at

α = 0.05

, five treatments, equal samples of size

n_{i} = 30

. Cases that were uninformative for comparing methods were omitted.

Distribution	Scale	Method
Distribution	${(σ}_{1} σ_{2} σ_{3} σ_{4} σ_{5})$	W50	OB	RMD	F₃	F₂	L₃	L₂	T₃	T₂
g = 0, h = 0	11111	0.044	0.023	0.045	0.041	0.047	0.042	0.047	0.038	0.041
g = 0, h = 0.4	11111	0.007	0.001	0.060	0.014	0.030	0.012	0.028	0.030	0.034
	31111	0.276	0.076	0.419	0.318	0.392	0.272	0.395	0.372	0.389
	32111	0.225	0.050	0.450	0.309	0.402	0.242	0.399	0.384	0.401
	51111	0.619	0.224	0.702	0.643	0.716	0.585	0.715	0.681	0.692
	53111	0.461	0.134	0.730	0.601	0.706	0.512	0.704	0.684	0.698
g = 0, h = 0.8	11111	0.002	0.000	0.075	0.013	0.029	0.009	0.019	0.033	0.043
	31111	0.026	0.007	0.155	0.074	0.112	0.053	0.097	0.116	0.132
	32111	0.023	0.003	0.199	0.070	0.124	0.042	0.103	0.143	0.167
	51111	0.078	0.015	0.278	0.183	0.241	0.124	0.228	0.241	0.252
	53111	0.059	0.008	0.339	0.187	0.274	0.110	0.245	0.289	0.299
g = 0.4, h = 0	11111	0.029	0.020	0.049	0.037	0.044	0.034	0.044	0.033	0.040
g = 0.4, h = 0.4	11111	0.004	0.001	0.073	0.014	0.030	0.011	0.022	0.035	0.045
	31111	0.222	0.068	0.380	0.280	0.349	0.228	0.346	0.332	0.352
	32111	0.168	0.041	0.409	0.264	0.349	0.199	0.340	0.338	0.360
	51111	0.533	0.174	0.635	0.561	0.660	0.506	0.664	0.613	0.628
	53111	0.394	0.099	0.671	0.546	0.642	0.448	0.636	0.613	0.634
g = 0.8, h = 0	11111	0.026	0.016	0.053	0.034	0.038	0.034	0.039	0.039	0.043
	31111	0.935	0.793	0.921	0.917	0.942	0.903	0.943	0.913	0.923
	32111	0.861	0.584	0.912	0.865	0.919	0.852	0.923	0.862	0.892
g = 0.8, h = 0.4	11111	0.005	0.001	0.084	0.012	0.035	0.009	0.031	0.036	0.050
	31111	0.138	0.046	0.303	0.215	0.265	0.170	0.263	0.256	0.276
	32111	0.103	0.024	0.336	0.200	0.274	0.154	0.262	0.267	0.296
	51111	0.371	0.118	0.526	0.456	0.514	0.392	0.517	0.490	0.507
	53111	0.264	0.067	0.573	0.436	0.533	0.325	0.528	0.510	0.531

Table 3. Proportion of at least one rejection at

α = 0.05