Robust and Nonrobust Linking of Two Groups for the Rasch Model with Balanced and Unbalanced Random DIF: A Comparative Simulation Study and the Simultaneous Assessment of Standard Errors and Linking Errors with Resampling Techniques

Abstract

In this article, the Rasch model is used for assessing a mean difference between two groups for a test of dichotomous items. It is assumed that random differential item functioning (DIF) exists that can bias group differences. The case of balanced DIF is distinguished from the case of unbalanced DIF. In balanced DIF, DIF effects on average cancel out. In contrast, in unbalanced DIF, the expected value of DIF effects can differ from zero and on average favor a particular group. Robust linking methods (e.g., invariance alignment) aim at determining group mean differences that are robust to the presence of DIF. In contrast, group differences obtained from nonrobust linking methods (e.g., Haebara linking) can be affected by the presence of a few DIF effects. Alternative robust and nonrobust linking methods are compared in a simulation study under various simulation conditions. It turned out that robust linking methods are preferred over nonrobust alternatives in the case of unbalanced DIF effects. Moreover, the theory of M-estimation, as an important approach to robust statistical estimation suitable for data with asymmetric errors, is used to study the asymptotic behavior of linking estimators if the number of items tends to infinity. These results give insights into the asymptotic bias and the estimation of linking errors that represent the variability in estimates due to selecting items in a test. Moreover, M-estimation is also used in an analytical treatment to assess standard errors and linking errors simultaneously. Finally, double jackknife and double half sampling methods are introduced and evaluated in a simulation study to assess standard errors and linking errors simultaneously. Half sampling outperformed jackknife estimators for the assessment of variability of estimates from robust linking methods.

1. Introduction

The analysis of psychological or educational tests is an important field in the social sciences. The test items (i.e., tasks presented in these tests) are often analyzed using item response theory (IRT, [1,2]) models. In this article, the Rasch model (RM; [3,4]) is used for comparing two groups on test items. For example, groups could be demographic groups, countries, studies, or time points. The group comparisons are carried out using linking methods [5,6]. An important impediment in applying linking methods is that the items could behave differently in the two groups (i.e., differential item functioning, DIF; [7]); that is, it cannot be expected that the Rasch model holds in the two groups with item parameters that are independent of a group membership.

In this article, we study the performance of linking methods in the presence of DIF that can bias group differences. In contrast to habitually used (i.e., nonrobust) linking methods, robust linking methods aim at deriving estimates of group differences that are robust to the presence of DIF. Importantly, DIF effects can be considered as asymmetric error distributions, and robust statistical methods for location measures are applied for determining a group difference.

This article systematically compares alternative linking methods in the RM. Furthermore, linking errors that quantify the uncertainty in group differences due to the randomness associated with DIF are analytically treated using M-estimation theory and computationally assessed using single and double jackknife and (balanced) half sampling, respectively.

The paper is structured as follows. In Section 2, the RM with random DIF is introduced. In Section 3, several nonrobust and robust linking methods are discussed. In Section 4, M-estimation theory is applied to the study of linking methods for the statistical inference of linking errors. In Section 5, M-estimation theory is applied for the simultaneous assessment of standard errors and linking errors. The resampling techniques double jackknife and double half sampling are introduced in Section 6 for empirically assessing standard errors and linking errors. In Section 7, we present the results of a simulation study in which different robust and nonrobust linking methods are systematically compared across various data-generating models for DIF effects. Section 8 presents a simulation study that investigates the empirical performance of the proposed resampling estimators from Section 6. Finally, the article concludes with a discussion in Section 9.

2. Differential Item Functioning in the Rasch Model

2.1. Rasch Model

The RM [3,4,8,9,10,11,12,13] is a statistical model for dichotomous item responses $X_i$ for items $i=1,\dots ,I$. A latent variable $\theta$ (so-called ability) accounts for the dependence among item responses. The item response function (IRF) $P_i$ for item i in the RM is defined as

$$\mathrm{P}(X_i=1|\theta )=P_i(\theta ;b_i)=\mathsf{\Psi }(\theta -b_i),$$

(1)

where $b_i$ is the item difficulty, $\theta$ is the latent ability, and $\mathsf{\Psi }\left(x\right)={[1+exp(-x)]}^{-1}$ denotes the logistic link function.

The RM in a random sampling perspective [14,15] also relies on a local independence assumption and poses a parametric distribution function $F_{\mathit{\alpha }}$ on the latent ability $\theta$:

$$\mathrm{P}(\mathit{X}=\mathit{x};\mathit{b},\mathit{\alpha })=P(\mathit{x};\mathit{b},\mathit{\alpha })=\int \left[\prod _{i=1}^I{P}_i{(\theta ;b_i)}^{x_i}{(1-P_i(\theta ;b_i))}^{1-x_i}\right]\mathrm{d}{F}_{\mathit{\alpha }}\left(\theta \right),$$

(2)

where $\mathit{X}=(X_1,\dots ,X_I)$, $\mathit{x}=(x_1,\dots ,x_I)$, $\mathit{b}=(b_1,\dots ,b_I)$ and a finite-dimensional parameter $\mathit{\alpha }$. In many applications, the distribution $F_{\mathit{\alpha }}$ is assumed to be normal with a mean of zero. In this case, the parameter $\mathit{\alpha }$ only contains the standard deviation $\sigma$ that must be estimated in addition to item parameters $\mathit{b}$. It has been empirically shown that distributional misspecifications of $F_{\mathit{\alpha }}$ might not strongly bias estimates of item difficulties $b_i$ if many items are available [16,17,18]. However, in (very) short tests and with a strong deviation from normality, the bias in item parameter estimates can be non-negligible.

Note that the parameters of the RM are identified up to a constant. Hence, either the mean of the abilities or the mean of item difficulties has to be fixed to zero for reasons of identification [19,20]. If a normal distribution for $\theta$ is assumed, the mean is set to zero, and only the standard deviation is estimated. The RM is typically estimated with marginal maximum likelihood estimation [21].

In practice, it is unlikely that the distribution of item responses can be adequately represented by the RM. In real datasets, more complex IRFs might be necessary, such as the family of logistic IRT models with two, three or four parameters per item [22] or even more flexible nonparametric monotone IRFs [23,24,25,26]. In large-scale educational datasets, items could have different discriminations [27], and guessing and slipping behavior have been reported [28,29]. However, fitting a misspecified RM to data might be justified if the latent ability $\theta$ should be defined such that all items contribute equally to $\theta$ [30,31]. By fitting more complex IRT models, the meaning of $\theta$ might change, which raises validity concerns. Note that the RM has been used in the popular program for international student assessment (PISA) study in the past [32] and in many other national [33,34] and international large-scale assessment studies [35].

The estimated item parameters $\widehat{\mathit{b}}$ with marginal maximum likelihood estimation [21] can be interpreted as a pseudo-true parameter (see [36]) that maximizes the Kullback–Leibler information $KL$ [37,38,39] between the true distribution Q and its parametric approximation $P(\mathit{b},\sigma )$:

$$(\widehat{\mathit{b}},\widehat{\sigma })=\underset{(\mathit{b},\sigma )}{arg\; max}KL\left(Q\right|P(\mathit{b},\sigma ))=\underset{(\mathit{b},\sigma )}{arg\; max}\sum _{s=1}^{S}Q\left({\mathit{x}}_s\right)logP({\mathit{x}}_s;\mathit{b},\sigma ),$$

(3)

where the sum is defined over the $S=2^I$ different item response patterns for $\mathit{X}$. The distribution Q is defined as the true data-generating distribution $Q\left(\mathit{x}\right)=P(\mathit{X}=\mathit{x})$, which is a multinomial distribution on the S item response patterns. The RM is the best approximation to Q with respect to the Kullback–Leibler information. By using different loss functions (i.e., estimation methods, for example, unweighted least squares estimation, [40]), different pseudo-true parameters of the RM will be obtained.

2.2. Differential Item Functioning

Now assume that I items are administered in two groups $g=1,2$. The estimation of group differences is of interest. Abilities in the first group follow a normal distribution with zero mean, that is, $\theta \sim \mathrm{N}(0,{\sigma }_1^2)$. In the second group, we also assume a normal distribution, i.e., $\theta \sim \mathrm{N}({\mu }_2,{\sigma }_2^2)$. The parameter ${\mu }_2$ can be interpreted as the average difference between the two groups.

In practical applications, it is unlikely that item difficulties $b_i$ are equal across groups (i.e., they are measurement invariant). In this case, DIF occurs, and there exist group-specific item difficulties $b_{ig}$ for groups $g=1,2$. Item-specific DIF effects $e_i$ are defined as $e_i=b_{i2}-b_{i1}$. In the absence of DIF, all DIF effects $e_i$ would be equal to zero. Identification constraints on DIF effects must be posed to disentangle group mean differences from average DIF effects [41,42,43].

In this paper, we distinguish the case of random items from fixed items with random DIF effects. In the first case of random items, it is assumed that the bivariate vector of item difficulties $(b_{i1},b_{i2})$ follows a bivariate distribution G. In the second case, it is assumed that $b_{i2}=b_{i1}+e_i$ with random effects $e_i$, but item difficulties $b_{i1}$ are regarded as fixed. This means that items are fixed, but DIF effects represent a random variable. DIF effects $e_i$ follow a univariate distribution G in this case.

To identify the group difference ${\mu }_2$, identification constraints on G have to be imposed in both cases. The main idea is that the set of items $\mathcal{I}=\{1,\dots ,I\}$ can be partitioned into two distinct sets ${\mathcal{I}}_{\mathrm{AMI}}$ and ${\mathcal{I}}_{\mathrm{bias}}$. The set of items in ${\mathcal{I}}_{\mathrm{AMI}}$ (also denoted as reference items; [44]) is deemed valid for obtaining unbiased group differences. The set ${\mathcal{I}}_{\mathrm{AMI}}$ refers to approximate measurement invariant (AMI; [45]) items. These items are allowed to have DIF effects that on average cancel out. A special case is a set of anchor items in which all items in this set have zero DIF effects [46]. Items in the set ${\mathcal{I}}_{\mathrm{bias}}$ (also denoted as biased items; [44]) have the potential to bias group differences (see [47]). The partitioning is modeled with a mixture distribution for G [46,48,49]:

$$G=(1-{\pi }_{\mathrm{bias}})G_{\mathrm{AMI}}+{\pi }_{\mathrm{bias}}{G}_{\mathrm{bias}},$$

(4)

where ${\pi }_{\mathrm{bias}}$ is the proportion of items in the set ${\mathcal{I}}_{\mathrm{bias}}$. In the fixed items case, it is assumed that the expected value for DIF effects of items from ${\mathcal{I}}_{\mathrm{AMI}}$ is zero, while it can be different from zero for items from ${\mathcal{I}}_{\mathrm{bias}}$. More formally, it holds that

$${\delta }_{\mathrm{AMI}}=\int e\mathrm{d}{G}_{\mathrm{AMI}}\left(e\right)=0\mathrm{and}{\delta }_{\mathrm{bias}}=\int e\mathrm{d}{G}_{\mathrm{bias}}\left(e\right).$$

(5)

In the random items case, define by $G^{*}$ the univariate distribution of DIF effects $e_i=b_{i2}-b_{i1}$. Based on the mixture representation of the bivariate distribution G, one can decompose the distribution of DIF effects as $G^{*}=(1-{\pi }_{\mathrm{bias}})G_{\mathrm{AMI}}^{*}+{\pi }_{\mathrm{bias}}{G}_{\mathrm{bias}}^{*}$. The condition for DIF effects in the random items case is the same as in Equation 5

$${\delta }_{\mathrm{AMI}}=\int e\mathrm{d}{G}_{\mathrm{AMI}}^{*}\left(e\right)=0\mathrm{and}{\delta }_{\mathrm{bias}}=\int e\mathrm{d}{G}_{\mathrm{bias}}^{*}\left(e\right).$$

(6)

The test is said to have balanced DIF if ${\delta }_{\mathrm{bias}}=0$, and it has unbalanced DIF if ${\delta }_{\mathrm{bias}}\ne 0$ (see [47,50,51]). It is important to emphasize that the definition of the mixture distribution allows the identification of group differences. The total DIF impact $\delta$ on the test containing all items can be calculated as (for notational simplicity only in the fixed items case)

$$\delta =\int e\mathrm{d}G\left(e\right)={\pi }_{\mathrm{bias}}{\delta }_{\mathrm{bias}}.$$

(7)

With a low proportion ${\pi }_{\mathrm{bias}}$ of biased items, the presence of DIF effects is not expected to have a large impact on estimated group differences.

In this article, we only consider random DIF effects. Probably in the largest part of the literature, DIF effects are considered as fixed (e.g., [52]). In this case, the condition for balanced DIF replaces the expected value by the mean associated with the fixed item parameters [44]. There is no additional uncertainty introduced in the estimation of group differences with fixed DIF effects because the item parameters are held fixed in repeated sampling. In contrast, with random DIF effects, the group mean difference is affected by the sampled DIF effects even for infinite sample sizes of persons. This kind of uncertainty is explicitly addressed in this article.

In many applications, the estimation of group differences involves a previous step in which DIF is detected by applying statistical techniques [7,52,53,54]. DIF detection statistics aim to classify items into a set of items that possess DIF, which should be optimally equal to the set ${\mathcal{I}}_{\mathrm{bias}}$. However, DIF detection techniques rely on previous knowledge about DIF-free items or a known group difference [43,55]. Hence, the decision of whether an item has DIF or not requires additional assumptions that cannot be statistically tested (see also [31,56,57]). In this paper, we do not thoroughly investigate DIF detection techniques, but rather study the performance of linking methods to estimate group differences. We distinguish robust from nonrobust linking methods. Robust linking methods adequately handle the presence of biased items (i.e., items in the set of ${\mathcal{I}}_{\mathrm{bias}}$) that lead to unbalanced DIF, while nonrobust linking approaches result in biased estimates of group differences in unbalanced DIF situations.

If the RM does not hold, DIF between groups means that IRFs can differ across the two groups. If the misspecified RM is fitted to data, DIF in item difficulties can be interpreted as a summary of DIF between IRFs. It is acknowledged that more complex DIF, such as nonuniform DIF in item discriminations [7] or DIF in guessing parameters, might occur. However, if these model aspects are intentionally ignored by fitting the RM, DIF effects in other aspects of the IRFs only indirectly enter the DIF assessment through item difficulties. Moreover, DIF effects in item difficulties are more frequently found in empirical applications than in item discriminations [58,59,60]. In the rest of the paper, statistical inference regarding the population of persons and the population of items is discussed that is even valid if the fitted IRT model is misspecified.

2.3. Identified Item Parameters in Group-Specific Scaling Models

Linking methods rely on group-specific item parameters estimated in separate scaling models in each group. By doing so, there is no misspecification in the IRT model due to noninvariance.

In the first group, the ability variable in the data-generating model follows $\theta \sim \mathrm{N}(0,{\sigma }_1^2)$, that is, ${\mu }_1=0$. In a separate estimation for the first group with an infinite sample size of persons, the estimated item difficulties ${\widehat{b}}_{i1}$ equal the data-generating parameters $b_{i1}$ (i.e., ${\widehat{b}}_{i1}=b_{i1}$). In the second group, the distribution of the ability variable is $\theta \sim \mathrm{N}({\mu }_2,{\sigma }_2^2)$. In the estimation, the mean of the ability variable is fixed to zero for reasons of identification. Hence, estimated item difficulties also include the group difference ${\mu }_2$ parameters. We obtain

$$\theta -b_{i2}={\sigma }_2{\theta }^{*}+{\mu }_2-b_{i2}={\theta }^{*}-{\sigma }_2^{-1}(b_{i2}-{\mu }_2)),$$

(8)

where the standardized ability ${\theta }^{*}$ is normally distributed (i.e., $\mathrm{N}(0,{\sigma }_2^2)$). Consequently, it follows that ${\widehat{b}}_{i2}=b_{i2}-{\mu }_2$.

3. Linking Methods

In this section, we review several linking methods [5,6,61,62,63] that allow the estimation of the group difference ${\mu }_2$. We assume that estimated identified item parameters ${\widehat{b}}_{i1}$ and ${\widehat{b}}_{i2}$ ($i=1,\dots ,I$) are available (see Section 2.3). We define differences ${\nu }_i={\widehat{b}}_{i1}-{\widehat{b}}_{i2}$.

3.1. Mean-Mean Linking (MM)

Mean-mean linking (MM; [5,64]) is one of the most popular linking methods. The group difference ${\mu }_2$ is estimated by

$${\widehat{\mu }}_2=\frac{1}{I}\sum _{i=1}^I{\widehat{b}}_{i1}-\frac{1}{I}\sum _{i=1}^I{\widehat{b}}_{i2}=\frac{1}{I}\sum _{i=1}^I{\nu }_i.$$

(9)

Note that ${\widehat{\mu }}_2$ is determined as the least-squares estimate of item difficulty differences ${\widehat{b}}_{i1}-{\widehat{b}}_{i2}$:

$${\widehat{\mu }}_2=\underset{{\mu }_2}{arg\; min}\left\{\frac{1}{I}\sum _{i=1}^I{({\mu }_2-{\nu }_i)}^2\right\}.$$

(10)

We now derive the bias of MM and assume fixed items with random DIF effects e that follow a distribution G. The distribution is given by the mixture representation $G=(1-{\pi }_{\mathrm{bias}})G_{\mathrm{AMI}}+{\pi }_{\mathrm{bias}}{G}_{\mathrm{bias}}$ (see Equation (4)). It holds that $\int e\mathrm{d}{G}_{\mathrm{AMI}}\left(e\right)=0$ and ${\delta }_{\mathrm{bias}}=\int e\mathrm{d}{G}_{\mathrm{bias}}\left(e\right)$ (see Equation (5)). Then, we obtain for the bias under MM

$$\begin{array}{ccc}\hfill \mathrm{Bias}\left({\widehat{\mu }}_2\right)& =& {\displaystyle -\frac{1}{I}\sum _{i=1}^I\mathrm{E}\left(e_i\right)}\hfill \\ \hfill & =& {\displaystyle -(1-{\pi }_{\mathrm{bias}})\int e\mathrm{d}{G}_{\mathrm{AMI}}\left(e\right)-{\pi }_{\mathrm{bias}}\int e\mathrm{d}{G}_{\mathrm{bias}}\left(e\right)}\hfill \\ & =& -{\pi }_{\mathrm{bias}}{\delta }_{\mathrm{bias}}\hfill \end{array}$$

(11)

The bias coincides with the DIF impact on the test (see Equation (7)). The bias vanishes in the case of balanced DIF (i.e., ${\delta }_{\mathrm{bias}}=0$) or in the absence of biased items (${\pi }_{\mathrm{bias}}=0$). MM can be considered as a nonrobust linking method because biased items can affect the estimated group difference. As an alternative to such a nonrobust approach, it may be recommended to use linking methods based on robust statistical methodology [65] designed for resistant estimation under contamination (especially for data contaminated by outlying values). The following linking methods realize some kind of robustness against the presence of biased DIF items.

3.2. Asymmetrically Trimmed Mean (ATR)

An intuitive idea borrowed from robust statistics [66,67,68] is to consider biased DIF items as outliers [69,70] in estimating the location measure that is given as the group difference. Hence, robust alternatives to the mean (i.e., MM linking) can be established.

The asymmetrically trimmed mean (ATR) removes items with large differences from the estimation. By defining a trimming proportion $\alpha$, the ATR linking estimate ${\widehat{\mu }}_2$ is defined as the average of ${\nu }_i$ values for which the absolute differences $|{\nu }_i-\mathrm{mdn}\left(\nu \right)|$ are below the $(1-\alpha )$-quantile of these discrepancies. The main idea is that large discrepancies can be regarded as biased items and should be removed from group comparisons. The ATR estimate is formally defined as

$${\widehat{\mu }}_2=\frac{{\displaystyle \sum _{i=1}^I{\nu }_i{\mathbf{1}}_{\left\{{\nu }_i\right||{\nu }_i-\mathrm{mdn}\left(\nu \right)|\le q_{1-\alpha }(\nu -\mathrm{mdn}\left(\nu \right))\}}\left({\nu }_i\right)}}{{\displaystyle \sum _{i=1}^I{\mathbf{1}}_{\left\{{\nu }_i\right||{\nu }_i-\mathrm{mdn}\left(\nu \right)|\le q_{1-\alpha }(\nu -\mathrm{mdn}\left(\nu \right))\}}\left({\nu }_i\right)}},$$

(12)

where $q_{1-\alpha }$ denotes the $(1-\alpha )$-quantile, $\mathbf{1}$ the indicator function, and the median $\mathrm{mdn}$. The median $\mathrm{mdn}\left(\nu \right)$ instead of the mean $\overline{\nu }$ is used because the median will be typically more robust concerning outliers (i.e., biased DIF items). ATR linking has the potential to properly handle the situation of unbalanced DIF because it explicitly allows that there could only be biased items with unidirectional signs. The ATR estimator is related to the least trimmed absolute estimator [71,72], which is especially suitable for asymmetric contamination in the data. A similar idea of the ATR estimator is used in robust structural equation modeling for defining case weights used for downweighting outlying cases (see [73,74]). As an alternative to the ATR estimator, the least weighted squares estimator may be applied as a location estimator of location with high robustness as well as high efficiency [75].

3.3. Elimination of DIF Items with Subsequent Mean-Mean Linking (EL)

Another popular approach is that DIF items are from the group comparison. The identification of DIF items in the first step requires the definition of an appropriate statistic. In the simulation study, we assume that a preliminary group difference is estimated by the median $\mathrm{mdn}\left(\nu \right)$ of all differences ${\nu }_i$. An item is declared to have DIF if $|{\nu }_i-\mathrm{mdn}\left(\nu \right)|$ exceeds a prespecified cutoff K. In many studies, the mean instead of the median is used, and the corresponding condition is referred to as the equal-mean anchor [52]. However, using the median instead of the mean might be a more robust location estimate in the presence of DIF effects. The items with detected DIF are removed for the subsequent computation of MM linking [52]. More formally, the EL estimate can be written as

$${\widehat{\mu }}_2=\frac{{\displaystyle \sum _{i=1}^I{\nu }_i{\mathbf{1}}_{\left\{{\nu }_i\right||{\nu }_i-\mathrm{mdn}\left(\nu \right)|\le K\}}\left({\nu }_i\right)}}{{\displaystyle \sum _{i=1}^I{\mathbf{1}}_{\left\{{\nu }_i\right||{\nu }_i-\mathrm{mdn}\left(\nu \right)|\le K\}}\left({\nu }_i\right)}}.$$

(13)

The EL linking method by eliminating DIF items can be interpreted as another variant of a trimmed mean.

3.4. Bisquare Linking (BSQ)

Another robust estimate of the location parameter is based on the bisquare loss function $\rho$ (see [76]) that is defined by

$$\rho (x;K)=\left\{\begin{array}{cc}{\displaystyle \frac{K^2}{6}\left[1-{\left(1-\frac{x^2}{K^2}\right)}^3\right]}& \mathrm{if}\left|x\right|\le K\\ {\displaystyle \frac{K^2}{6}}& \mathrm{else}\end{array}\right.,$$

(14)

where K is a prespecified threshold value. The group difference ${\mu }_2$ is estimated by   

$${\widehat{\mu }}_2=\underset{{\mu }_2}{arg\; min}\left\{\frac{1}{I}\sum _{i=1}^I\rho ({\mu }_2-{\nu }_i;K)\right\}.$$

(15)

Note that the bisquare loss function is also known as the Tukey biweight function [77].

3.5. Invariance Alignment (IA)

The bisquare loss function $\rho$ in Equation (15) can be replaced by any robust (or nonrobust) loss function. In invariance alignment (IA; [78,79,80,81]), the $L_p$ power loss function $\rho \left(x\right)={\left|x\right|}^p$ ($p\in (0,2]$) is employed. The group mean estimate is given by

$${\widehat{\mu }}_2=\underset{{\mu }_2}{arg\; min}\left\{\frac{1}{I}\sum _{i=1}^I{\left|{\mu }_2-{\nu }_i\right|}^p\right\}.$$

(16)

By choosing $p\le 1$, the extent of noninvariant items is minimized. Hence, the group mean difference relies on items that have small DIF effects while removing items with large DIF effects from the comparison [78]. IA was originally proposed with the power $p=0.5$  [78]. IA with $p=2$ is equivalent to MM. Note that IA with $p<1$ is particularly suited to the situation of partial invariance in which $G_{\mathrm{AMI}}$ concentrates at zero (i.e., all DIF effects in ${\mathcal{I}}_{\mathrm{AMI}}$ are zero or close to zero) and fails for symmetrically distributed DIF effects [82,83].

3.6. Haebara Linking (HAE)

In contrast to MM, ATR, BSQ and IA linking methods, Haebara (HAE) linking [84] aligns IRFs instead of aligning item parameters. The linking function is defined as

$${\widehat{\mu }}_2=\underset{{\mu }_2}{arg\; min}\left\{\frac{1}{I}\sum _{i=1}^I\int {\left|\mathsf{\Psi }\left(\theta -{\widehat{b}}_{i1})\right)-\mathsf{\Psi }\left(\theta -{\widehat{b}}_{i2}-{\mu }_2)\right)\right|}^p\omega \left(\theta \right)\mathrm{d}\theta \right\}$$

(17)

with a power $p\ge 0$ and a weight function $\omega$ that fulfills $\int \omega \left(\theta \right)\mathrm{d}\theta =1$. The originally proposed HAE linking uses $p=2$ [84]. The robust alternative $p=1$ was treated in [47,85,86], while cases $p<1$ were studied in [87].

To get more insight into the relation of IA and HAE, we apply a Taylor approximation of the second IRF in Equation (17) under the assumption of small effects $e_i$. We obtain

$$\mathsf{\Psi }(\theta -{\widehat{b}}_{i2}-{\mu }_2)\simeq \mathsf{\Psi }(\theta -{\widehat{b}}_{i1})+{\mathsf{\Psi }}_1(\theta -{\widehat{b}}_{i1})({\mu }_2-{\nu }_i),$$

(18)

where ${\mathsf{\Psi }}_1\left(x\right)=\frac{\mathrm{d}\mathsf{\Psi }}{\mathrm{d}\theta }=\mathsf{\Psi }\left(x\right)(1-\mathsf{\Psi }\left(x\right))\ge 0$. Using the approximation (18), Equation (17) can be rewritten as

$${\widehat{\mu }}_2=\underset{{\mu }_2}{arg\; min}\left\{\frac{1}{I}\sum _{i=1}^I{w}_i{|{\mu }_2-{\nu }_i|}^p\right\},$$

(19)

where item-specific weights are given by $w_i=\int ({\mathsf{\Psi }}_1{(\theta -{\widehat{b}}_{i1})}^p\omega \left(\theta \right)\mathrm{d}\theta$. Hence, HAE linking can be interpreted as IA with item-specific weights, and a similar performance of HAE and IA can be expected.

3.7. Gini Linking (GI)

Recently, a linking procedure based on the Gini index (GI; [88]) has been proposed. The linking function is very similar to IA linking and tries to define a group difference that is primarily based on items with small DIF effects. The group mean difference is determined by

$${\widehat{\mu }}_2=\underset{{\mu }_2}{arg\; max}\left\{\frac{{\displaystyle \sum _{i=1}^I\sum _{j=1}^I\left||{\mu }_2-{\nu }_i{|}^p-{|{\mu }_2-{\nu }_i|}^p\right|}}{{\displaystyle 2I\sum _{i=1}^I{|{\mu }_2-{\nu }_i|}^p}}\right\},$$

(20)

where the power $p>0$ can be chosen by the user. The original proposal used $p=1$ [88]. Previous experience of the authors indicates that GI also works with $p>1$, but it does not perform satisfactorily with $p<1$. It has been shown that IA and GI provided similar results in small case studies [88], but GI linking has not yet been systematically compared with other linking methods.

3.8. Robustness of the Different Linking Methods

The linking methods mean-mean linking (MM) and Haebara linking (HAE) with $p=2$ can be considered as nonrobust. The linking methods based on the asymmetrically trimmed mean (ATR), elimination of DIF items with subsequent mean-mean linking (EL), bisquare linking (BSQ), invariance alignment (IA) with $p\le 1$, Haebara linking (HAE) with $p\le 1$ and Gini linking (GI) can be considered as robust linking methods that ensure some protection to the presence of biased DIF items.

4. An Analytical Treatment for Assessing Linking Errors

In this section, the computation of linking errors is investigated. Linking errors refer to the uncertainty of the randomness associated with items [32,81,89,90,91,92,93]. The estimated group difference ${\widehat{\mu }}_2$ is affected by random DIF. The linking error quantifies this source of variance. In this section, an analytical treatment for assessing linking errors is presented. For this section, we assume an infinite sample size of persons. That means that identified item parameters are estimated without a sampling error. This assumption is dropped in the next Section 5.

In Section 2.2, we assume that random DIF (or item parameters) follow(s) a mixture distribution $G=(1-{\pi }_{\mathrm{bias}})G_{\mathrm{AMI}}+{\pi }_{\mathrm{bias}}{G}_{\mathrm{bias}}$, where $G_{\mathrm{AMI}}$ denotes the distribution associated with reference items for which DIF effects on average cancel out and $G_{\mathrm{bias}}$ denotes the distribution of biased items that can impact estimated group differences. The estimation of the group difference can be interpreted as an estimation problem of a location parameter in robust statistics where the location parameter (i.e., the group difference ${\mu }_2$) should be based on $G_{\mathrm{AMI}}$. However, the observed mixture distribution G contains a contaminated asymmetric error distribution [67,94,95] that might bias the estimate ${\widehat{\mu }}_2$. As discussed in Section 2.2, two cases of random DIF can be distinguished. First, items can be considered random, and the bivariate vector $(b_{i1},b_{i2})$ of group-specific item difficulties is modeled with a distribution (see Section 4.1). Second, items can be regarded as fixed, but DIF effects $e_i=b_{i2}-b_{i1}$ are modeled as a random variable (see Section 4.2). Although these cases are very different, their consequences lead to similar estimates of variances. Hence, estimated errors (i.e., linking errors) due to the randomness associated with items are practically identical.

4.1. Random Item Parameters $(b_{i1},b_{i2})$

In this subsection, we discuss the estimation for random item parameters. We introduce a slightly more general notation to cover the linking methods (except for GI linking) from Section 3. The “data” for item i is given by the vector ${\mathit{y}}_i=({\widehat{b}}_{1i},{\widehat{b}}_{2i})=(b_{i1},b_{i2}+{\mu }_2)$. The linking method must be additive with respect to functions of this data. More formally, let H be a linking function that is defined by

$$H\left(\mathit{\gamma }\right)=\frac{1}{I}\sum _{i=1}^{I}h({\mathit{y}}_i,\mathit{\gamma }).$$

(21)

The linking parameter $\mathit{\gamma }$ (e.g., a group difference) of interest is estimated by

$$\widehat{\mathit{\gamma }}=\underset{\mathit{\gamma }}{arg\; min}H\left(\mathit{\gamma }\right).$$

(22)

For a large number of items I, note that $H\left(\mathit{\gamma }\right)$ defined in (21) converges to   

$$H_0(\mathit{\gamma };G)=\int h(\mathit{y},\mathit{\gamma })\mathrm{d}G\left(\mathit{y}\right).$$

(23)

Assuming differentiability of h implies that $\widehat{\mathit{\gamma }}$ can be obtained by solving the equation

$$\mathit{S}\left(\mathit{\gamma }\right)=\frac{\partial }{\partial \mathit{\gamma }}H\left(\mathit{\gamma }\right)=\frac{1}{I}\sum _{i=1}^I\frac{\partial }{\partial \mathit{\gamma }}h({\mathit{y}}_i,\mathit{\gamma })=\frac{1}{I}\sum _{i=1}^I\mathit{m}({\mathit{y}}_i,\mathit{\gamma })=\mathbf{0},$$

(24)

where $\mathit{m}=\frac{\partial h}{\partial \mathit{\gamma }}$.

Equation (24) provides an estimating equation for the parameter $\mathit{\gamma }$. The corresponding estimator is labeled as an M-estimator [96]. It is evident that the estimated group mean differences ${\widehat{\mu }}_2$ in MM, IA and HAE linking are M-estimators by defining the univariate parameter $\widehat{\mathit{\gamma }}=\left({\widehat{\mu }}_2\right)$. The linking methods EL and ATR are so-called two-step estimators because their computation relies on the median ${\tilde{\mu }}_2=\mathrm{mdn}\left(\nu \right)$ computed in a first step. Because the estimating equation for the median is clearly defined, the estimate in Equation (24) also applies to two-step estimators because it can be interpreted as a bivariate one-step M-estimator by defining $\widehat{\mathit{\gamma }}=({\tilde{\mu }}_2,{\widehat{\mu }}_2)$ (see [37], chp. 7).

We now apply the theory of M-estimators ([37], chp. 7; [96,97]) to study the asymptotic behavior of $\widehat{\mathit{\gamma }}$. Because we are concerned with linking errors, asymptotic behavior is meant with respect to the number of items. By letting the number of items tend to infinity, the left side in Equation (24) converges to

$${\mathit{S}}_0(\mathit{\gamma };G)=\int \mathit{m}(\mathit{y},\mathit{\gamma })\mathrm{d}G\left(\tilde{\mathit{y}}\right),$$

(25)

where $\tilde{\mathit{y}}=({\widehat{b}}_1,{\widehat{b}}_2-{\mu }_2)$ and ${\widehat{b}}_g$ ($g=1,2$) denote the random variables associated with estimated item parameters. As already mentioned, the distribution of item parameters G follows the mixture distribution $G=(1-{\pi }_{\mathrm{bias}})G_{\mathrm{AMI}}+{\pi }_{\mathrm{bias}}{G}_{\mathrm{bias}}$. Assume that densities for the involved distributions exist (i.e., continuous or count densities): $\mathrm{d}G=g\mathrm{d}\theta$, $\mathrm{d}{G}_{\mathrm{AMI}}=g_{\mathrm{AMI}}\mathrm{d}\theta$, and $\mathrm{d}{G}_{\mathrm{bias}}=g_{\mathrm{bias}}\mathrm{d}\theta$. Equation (25) can be written as

$${\mathit{S}}_0(\mathit{\gamma };G)=(1-{\pi }_{\mathrm{bias}}){\mathit{S}}_0(\mathit{\gamma },G_{\mathrm{AMI}})+{\pi }_{\mathrm{bias}}{\mathit{S}}_0(\mathit{\gamma },G_{\mathrm{bias}}).$$

(26)

The parameter ${\mathit{\gamma }}_{\infty }$ obtained from a linking method with an infinite number of items is given as the root of the equation

$${\mathit{S}}_0({\mathit{\gamma }}_{\infty };G)=\int \mathit{m}(\mathit{y},{\mathit{\gamma }}_{\infty })\mathrm{d}G\left(\tilde{\mathit{y}}\right)=\mathbf{0}.$$

(27)

Note that ${\mathit{\gamma }}_{\infty }$ is a function of ${\mu }_2$, G, and $\mathit{m}$. For a dataset, ${\mu }_2$ and G are fixed but unknown. However, $\mathit{m}$ is chosen in the linking method by a user.

The pseudo-true parameter ${\mathit{\gamma }}_{\mathrm{AMI}}$ is defined as the estimate if all items would be reference items. That is, the linking parameter $\mathit{\gamma }$ would only be determined by the mixture component $G_{\mathrm{AMI}}$:

$${\mathit{S}}_0({\mathit{\gamma }}_{\mathrm{AMI}};G_{\mathrm{AMI}})=\int \mathit{m}(\mathit{y},{\mathit{\gamma }}_{\mathrm{AMI}})\mathrm{d}{G}_{\mathrm{AMI}}\left(\tilde{\mathit{y}}\right)=\mathbf{0}.$$

(28)

Ideally, a component of ${\mathit{\gamma }}_{\mathrm{AMI}}$ (in the bivariate case) or ${\mathit{\gamma }}_{\mathrm{AMI}}$ itself (in the univariate case) should provide an asymptotically unbiased estimate of ${\mu }_2$ by choosing an appropriate linking function h (or its derivative $\mathit{m}$). In the following, we assume that $\mathit{m}$ is differentiable, although the main propositions of M-estimators do not require differentiability [37]. However, one could always approximate a nondifferentiable linking function h by a differentiable approximation $h_{\mathcal{A}}$. For example, the nondifferentiable and nonnegative linking function $h\left(x\right)$ can be approximated by $h_{\mathcal{A}}\left(x\right)=\sqrt{h{\left(x\right)}^2+\epsilon }$ for a sufficiently small $\epsilon >0$ [78,80,81,87].

4.1.1. Asymptotic Behavior

We now study the asymptotic behavior of the estimator $\widehat{\mathit{\gamma }}$. For a large number of items, $\widehat{\mathit{\gamma }}$ converges to ${\mathit{\gamma }}_{\infty }$. The derivation of ${\mathit{\gamma }}_{\infty }$ relies on a Taylor approximation of $\mathit{m}$ and closely follows [98]. Due to (26), we get

$${\mathit{S}}_0({\mathit{\gamma }}_{\infty };G)=(1-{\pi }_{\mathrm{bias}}){\mathit{S}}_0({\mathit{\gamma }}_{\infty },G_{\mathrm{AMI}})+{\pi }_{\mathrm{bias}}{\mathit{S}}_0({\mathit{\gamma }}_{\infty },G_{\mathrm{bias}})=\mathbf{0}.$$

(29)

We now apply a first-order Taylor approximation of $\mathit{m}$ around $\mathit{\gamma }={\mathit{\gamma }}_{\mathrm{AMI}}$:

$$\mathit{m}(\mathit{y},{\mathit{\gamma }}_{\infty })\simeq \mathit{m}(\mathit{y},{\mathit{\gamma }}_{\mathrm{AMI}})+{\mathit{M}}_{\mathit{\gamma }}(\mathit{y},{\mathit{\gamma }}_{\mathrm{AMI}})({\mathit{\gamma }}_{\infty }-{\mathit{\gamma }}_{\mathrm{AMI}}),$$

(30)

where ${\mathit{M}}_{\mathit{\gamma }}=\frac{\partial \mathit{m}}{\partial \mathit{\gamma }}$ is the matrix of partial derivatives. From (29) and (30), we get

$$\begin{array}{ccc}\hfill {\displaystyle (1-{\pi }_{\mathrm{bias}})\left[\int {\mathit{M}}_{\mathit{\gamma }}(\mathit{y},{\mathit{\gamma }}_{\mathrm{AMI}})\mathrm{d}{G}_{\mathrm{AMI}}\left(\tilde{\mathit{y}}\right)\right]({\mathit{\gamma }}_{\infty }-{\mathit{\gamma }}_{\mathrm{AMI}})}& & \\ \hfill +{\pi }_{\mathrm{bias}}\left[{\displaystyle \int \mathit{m}(\mathit{y},{\mathit{\gamma }}_{\infty })\mathrm{d}{G}_{\mathrm{bias}}\left(\tilde{\mathit{y}}\right)}\right]& =& \mathbf{0}\hfill \end{array}$$

(31)

Hence, we obtain from (31),

$${\mathit{\gamma }}_{\infty }-{\mathit{\gamma }}_{\mathrm{AMI}}=\frac{{\pi }_{\mathrm{bias}}}{1-{\pi }_{\mathrm{bias}}}{\left[\int {\mathit{M}}_{\mathit{\gamma }}(\mathit{y},{\mathit{\gamma }}_{\mathrm{AMI}})\mathrm{d}{G}_{\mathrm{AMI}}\left(\tilde{\mathit{y}}\right)\right]}^{-1}\left[{\displaystyle \int \mathit{m}(\mathit{y},{\mathit{\gamma }}_{\infty })\mathrm{d}{G}_{\mathrm{bias}}\left(\tilde{\mathit{y}}\right)}\right].$$

(32)

If we assume that ${\mathit{\gamma }}_{\mathrm{AMI}}$ allows the unbiased estimation of ${\mu }_2$, Equation (32) provides an expression of the asymptotic bias of $\widehat{\mathit{\gamma }}$. Of crucial importance is that the linking function $\mathit{m}$ downweights observations from the distribution of biased items $G_{\mathrm{bias}}$ (i.e., ${\displaystyle \int \mathit{m}(\mathit{y},{\mathit{\gamma }}_{\infty })\mathrm{d}{G}_{\mathrm{bias}}\left(\tilde{\mathit{y}}\right)\approx 0}$). The linking function $\mathit{m}$ has to be chosen so that biased items are automatically removed for group comparison. The next subsection discusses how the linking function $\mathit{m}$ should be chosen to enable an unbiased estimation of ${\mu }_2$.

4.1.2. Choosing an Optimal Linking Function m

Again, the derivation of the choice of the linking function $\mathit{m}$ follows the exposition in [98]. Assume that the true parameter ${\mu }_2$ is determined by the distribution $G_{\mathrm{AMI}}$ (with density $g_{\mathrm{AMI}}$) of reference items. Hence, ${\mu }_2$ is given as the maximizer of the log-likelihood function and fulfills

$${\displaystyle \mathrm{E}\left(\frac{\mathrm{d}}{\mathrm{d}{\mu }_2}log{g}_{\mathrm{AMI}}\left(\tilde{\mathit{y}}\right)\right)=\int \frac{\mathrm{d}}{\mathrm{d}{\mu }_2}log{g}_{\mathrm{AMI}}\left(\tilde{\mathit{y}}\right)g_{\mathrm{AMI}}\left(\tilde{\mathit{y}}\right)\mathrm{d}\tilde{\mathit{y}}=0.}$$

(33)

Based on (33), the linking function $\mathit{m}$ can be chosen in order to obtain unbiased estimates of group mean differences ${\mu }_2$ (see [98]):

$$\mathit{m}(\mathit{y},{\mu }_2)=\left(\frac{\mathrm{d}}{\mathrm{d}{\mu }_2}log{g}_{\mathrm{AMI}}\left(\tilde{\mathit{y}}\right)\right)w(\mathit{y},{\mu }_2)$$

(34)

with the weight function w defined as

$$w(\mathit{y},{\mu }_2)=\frac{(1-{\pi }_{\mathrm{bias}})g_{\mathrm{AMI}}\left(\tilde{\mathit{y}}\right)}{g\left(\tilde{\mathit{y}}\right)}={\left[1+\frac{{\pi }_{\mathrm{bias}}{g}_{\mathrm{bias}}\left(\tilde{\mathit{y}}\right)}{(1-{\pi }_{\mathrm{bias}})g_{\mathrm{AMI}}\left(\tilde{\mathit{y}}\right)}\right]}^{-1}.$$

(35)

Note that $0<w(\mathit{y},{\mu }_2)\le 1$ and the weighting function w weighs observations $\mathit{y}$ according to their closeness to the distribution $G_{\mathrm{AMI}}$. Observations $\mathit{y}$ with large density values $g_{\mathrm{bias}}\left(\tilde{\mathit{y}}\right)$ are downweighted in w. Using (33), it can be shown that

$${\displaystyle \int \mathit{m}(\mathit{y},{\mu }_2)g\left(\tilde{\mathit{y}}\right)\mathrm{d}\tilde{\mathit{y}}=0.}$$

(36)

4.1.3. Asymptotic Normal Distribution

We now show that the M-estimator $\widehat{\mathit{\gamma }}$ follows an asymptotic normal (AN) distribution (see [37], chp. 7). The same Taylor expansion as in (30) provides

$$\mathit{m}(\mathit{y},\widehat{\mathit{\gamma }})\simeq \mathit{m}(\mathit{y},{\mathit{\gamma }}_{\infty })+{\mathit{M}}_{\mathit{\gamma }}(\mathit{y},{\mathit{\gamma }}_{\infty })(\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_{\infty }).$$

(37)

The approximation (37) can be substituted into the estimating Equation (24):

$$\mathbf{0}=\frac{1}{I}\sum _{i=1}^I\mathit{m}({\mathit{y}}_i,\mathit{\gamma })\simeq \frac{1}{I}\sum _{i=1}^I\mathit{m}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })+\left(\frac{1}{I}\sum _{i=1}^I{\mathit{M}}_{\mathit{\gamma }}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })\right)(\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_{\infty }).$$

(38)

Hence, we obtain from (38)

$$\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_{\infty }\simeq -{\left(\frac{1}{I}\sum _{i=1}^I{\mathit{M}}_{\mathit{\gamma }}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })\right)}^{-1}\left(\frac{1}{I}\sum _{i=1}^I\mathit{m}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })\right).$$

(39)

For $I\to \infty$, we have (see (24))

$$\frac{1}{I}\sum _{i=1}^I{\mathit{M}}_{\mathit{\gamma }}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })\stackrel{p}{\to }\mathit{A}\left({\mathit{\gamma }}_{\infty }\right)=-\int {\mathit{M}}_{\mathit{\gamma }}(\mathit{y},{\mathit{\gamma }}_{\infty })\mathrm{d}G\left(\tilde{\mathit{y}}\right)\mathrm{and}$$

(40)

$$\frac{1}{I}\sum _{i=1}^I\mathit{m}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })\stackrel{p}{\to }{\mathit{S}}_0({\mathit{\gamma }}_{\infty },G)=\int \mathit{m}(\mathit{y},{\mathit{\gamma }}_{\infty })\mathrm{d}G\left(\tilde{\mathit{y}}\right)=\mathbf{0}.$$

(41)

Therefore, we obtain the asymptotic normal distribution of $\widehat{\mathit{\gamma }}$ as

$$\widehat{\mathit{\gamma }}\sim \mathrm{AN}\left({\mathit{\gamma }}_{\infty },\frac{1}{I}\mathit{A}{\left({\mathit{\gamma }}_{\infty }\right)}^{-1}\mathit{B}\left({\mathit{\gamma }}_{\infty }\right)\mathit{A}{\left({\mathit{\gamma }}_{\infty }\right)}^{-\top }\right),\mathrm{where}$$

(42)

$$\mathit{B}\left({\mathit{\gamma }}_{\infty }\right)=\int \mathit{m}(\mathit{y},{\mathit{\gamma }}_{\infty })\mathit{m}{(\mathit{y},{\mathit{\gamma }}_{\infty })}^T\mathrm{d}G\left(\tilde{\mathit{y}}\right).$$

(43)

The involved matrices $\mathit{A}\left({\mathit{\gamma }}_{\infty }\right)$ and $\mathit{B}\left({\mathit{\gamma }}_{\infty }\right)$ can be estimated from sample data by

$$\widehat{\mathit{A}}\left(\widehat{\mathit{\gamma }}\right)=-\frac{1}{I}\sum _{i=1}^I{\mathit{M}}_{\mathit{\gamma }}({\mathit{y}}_i,\widehat{\mathit{\gamma }})\mathrm{and}$$

(44)

$$\widehat{\mathit{B}}\left(\widehat{\mathit{\gamma }}\right)=\frac{1}{I}\sum _{i=1}^I\mathit{m}({\mathit{y}}_i,\widehat{\mathit{\gamma }})\mathit{m}{({\mathit{y}}_i,\widehat{\mathit{\gamma }})}^{\top }.$$

(45)

Notably, the distribution stated in Equation (42) only holds for a sufficiently large number of items I.

4.1.4. Scalar Linking Parameter

We now specialize our results if the estimated parameter $\widehat{\mathit{\gamma }}$ coincides with the estimated group difference ${\widehat{\mu }}_2$. In this case, m is a univariate linking function. Assume that ${\mu }_{2,\mathrm{AMI}}$ and ${\mu }_{2,\infty }$ are the roots of the following equations, respectively:

$${\displaystyle \int m(\mathit{y},{\mu }_{2,\mathrm{AMI}})\mathrm{d}{G}_{\mathrm{AMI}}\left(\tilde{\mathit{y}}\right)=0\mathrm{and}}$$

(46)

$${\displaystyle \int m(\mathit{y},{\mu }_{2,\infty })\mathrm{d}G\left(\tilde{\mathit{y}}\right)=0.}$$

(47)

The asymptotic behavior of ${\widehat{\mu }}_2$ can be described as (see Equation (32))   

$${\mu }_{2,\infty }-{\mu }_{2,\mathrm{AMI}}=-\frac{{\pi }_{\mathrm{bias}}}{1-{\pi }_{\mathrm{bias}}}\frac{{\displaystyle \int m(\mathit{y},{\mu }_{2,\infty })\mathrm{d}{G}_{\mathrm{bias}}\left(\tilde{\mathit{y}}\right)}}{{\displaystyle \int m^{\prime }(\mathit{y},{\mu }_{2,\mathrm{AMI}})\mathrm{d}{G}_{\mathrm{AMI}}\left(\tilde{\mathit{y}}\right)}},$$

(48)

where $m^{\prime }$ is the derivative of m with respect to ${\mu }_2$. Furthermore, ${\widehat{\mu }}_2$ is asymptotically normally distributed (see Equation (42)):

$${\widehat{\mu }}_2\sim \mathrm{AN}\left({\mu }_{2,\infty },\frac{1}{I}\frac{{\displaystyle \int {\left\{m(\mathit{y},{\mu }_{2,\infty })\right\}}^2\mathrm{d}G\left(\mathit{y}\right)}}{{\displaystyle {\left\{\int m^{\prime }(\mathit{y},{\mu }_{2,\infty })\mathrm{d}G\left(\mathit{y}\right)\right\}}^2}}\right).$$

(49)

Again, the involved integrals for the variance estimate in (49) can be estimated using sample data (see Equations (44) and (45)).

4.2. Fixed Item Parameters $b_i$, Random DIF Effects $e_i$

In this subsection, we consider the case of fixed item parameters $b_i$, but DIF effects $e_i$ are random. The “data” in Section 4.1 was given by ${\mathit{y}}_i=({\widehat{b}}_{1i},{\widehat{b}}_{2i})=(b_i,-{\mu }_2+b_i+e_i)$, and DIF effects $e_i$ follow a distribution G. Now, only $e_i$ is random and we define the data as $y_i={\widehat{b}}_{2i}-{\widehat{b}}_{1i}=-{\mu }_2+e_i$ and ${\tilde{y}}_i=y_i+{\mu }_2$.

The estimating Equation (24) for the linking parameter $\mathit{\gamma }$ can be rewritten as

$$\mathit{S}\left(\mathit{\gamma }\right)=\frac{1}{I}\sum _{i=1}^I\mathit{m}(b_i,y_i,\mathit{\gamma })=\mathbf{0}.$$

(50)

The term in (50) converges to

$${\tilde{\mathit{S}}}_0(\mathit{\gamma };G)=\underset{I\to \infty }{lim}\frac{1}{I}\sum _{i=1}^I\int \mathit{m}(b_i,y_i,\mathit{\gamma })\mathrm{d}G\left({\tilde{y}}_i\right)=\mathbf{0}.$$

(51)

One has to assume that ${\tilde{\mathit{S}}}_0(\mathit{\gamma };G)$ exists. Define ${\mathit{\gamma }}_{\infty }$ by

$${\tilde{\mathit{S}}}_0({\mathit{\gamma }}_{\infty };G)=\underset{I\to \infty }{lim}\frac{1}{I}\sum _{i=1}^I\int \mathit{m}(b_i,y_i,{\mathit{\gamma }}_{\infty })\mathrm{d}G\left({\tilde{y}}_i\right)=\mathbf{0}.$$

(52)

Then, we can derive an asymptotic normal distribution for $\widehat{\mathit{\gamma }}$:

$$\widehat{\mathit{\gamma }}\sim \mathrm{AN}\left({\mathit{\gamma }}_{\infty },\frac{1}{I}\tilde{\mathit{A}}{\left({\mathit{\gamma }}_{\infty }\right)}^{-1}\tilde{\mathit{B}}\left({\mathit{\gamma }}_{\infty }\right)\tilde{\mathit{A}}{\left({\mathit{\gamma }}_{\infty }\right)}^{-\top }\right),\mathrm{where}$$

(53)

$$\tilde{\mathit{A}}\left({\mathit{\gamma }}_{\infty }\right)=-\underset{I\to \infty }{lim}\frac{1}{I}\sum _{i=1}^I\int {\mathit{M}}_{\mathit{\gamma }}(b_i,y_i,{\mathit{\gamma }}_{\infty })\mathrm{d}G\left({\tilde{y}}_i\right)$$

(54)

$$\tilde{\mathit{B}}\left({\mathit{\gamma }}_{\infty }\right)=\underset{I\to \infty }{lim}\frac{1}{I}\sum _{i=1}^I\int \mathit{m}(b_i,y_i,{\mathit{\gamma }}_{\infty })\mathit{m}{(b_i,y_i,{\mathit{\gamma }}_{\infty })}^{\top }\mathrm{d}G\left({\tilde{y}}_i\right)$$

(55)

The involved matrices $\tilde{\mathit{A}}\left({\mathit{\gamma }}_{\infty }\right)$ and $\tilde{\mathit{B}}\left({\mathit{\gamma }}_{\infty }\right)$ can be estimated by

$$\widehat{\mathit{A}}\left(\widehat{\mathit{\gamma }}\right)=-\frac{1}{I}\sum _{i=1}^I{\mathit{M}}_{\mathit{\gamma }}(b_i,y_i,\widehat{\mathit{\gamma }})\mathrm{and}$$

(56)

$$\widehat{\mathit{B}}\left(\widehat{\mathit{\gamma }}\right)=\frac{1}{I}\sum _{i=1}^I\mathit{m}(b_i,y_i,\widehat{\mathit{\gamma }})\mathit{m}{(b_i,y_i,\widehat{\mathit{\gamma }})}^{\top }.$$

(57)

Interestingly, these estimators coincide with estimated standard errors in the case of random item parameters (see Equations (44) and (45)). Hence, no practical differences regarding the estimated linking parameters and their estimated standard errors can be expected. Only conceptual differences emerge for the two treatments of DIF effects.

5. An Analytical Treatment for the Simultaneous Assessment of Standard Errors and Linking Errors

In practice, the variance in the group mean difference is affected by the sampling of persons (i.e., standard error) and the randomness associated with items (i.e., linking error). There have been attempts for the analytical treatment of the simultaneous inference with respect to the two modes [81,99,100]. In this section, we apply M-estimation theory for the simultaneous assessment of standard errors and linking errors. The general idea in this kind of inference is investigating the asymptotic behavior of the M-estimator $\widehat{\mathit{\gamma }}$ if the number of persons P and the number of items I tend to infinity. We only consider the case of random items, but treatment of the case with fixed items and random DIF effects is similar.

In the notation of Section 4, ${\mathit{y}}_i$ denotes the vector of (true) identified item parameters. In finite samples of size P, only estimates ${\widehat{\mathit{y}}}_i$ are available. For $P\to \infty$, it holds that ${\widehat{\mathit{y}}}_i\stackrel{p}{\to }{\mathit{y}}_i$. In long tests, the estimated item parameters are approximately independent between items [101]. Hence, we can assume that ${\widehat{\mathit{y}}}_i$ are approximately independent of each other. M-estimation theory applied to the person side guarantees an asymptotic normal distribution:

$${\widehat{\mathit{y}}}_i\sim \mathrm{AN}\left({\mathit{y}}_i,\frac{1}{P}{\mathit{V}}_i\left({\mathit{y}}_i\right)\right),$$

(58)

where ${\mathit{V}}_i\left({\mathit{y}}_i\right)$ is a function of true item parameters ${\mathit{y}}_i$. We now use a Taylor expansion with respect to $\mathit{\gamma }$ and ${\mathit{y}}_i$

$$\mathit{m}({\widehat{\mathit{y}}}_i,\widehat{\mathit{\gamma }})\simeq \mathit{m}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })+{\mathit{M}}_{\mathit{\gamma }}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })(\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_{\infty })+{\mathit{M}}_{\mathit{y}}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })({\widehat{\mathit{y}}}_i-{\mathit{y}}_i).$$

(59)

Using the same approach as in Section 4.1.3, we get an approximation of the estimating equation as

$$\frac{1}{I}\sum _{i=1}^I\left[\mathit{m}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })+{\mathit{M}}_{\mathit{\gamma }}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })(\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_{\infty })+{\mathit{M}}_{\mathit{y}}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })({\widehat{\mathit{y}}}_i-{\mathit{y}}_i)\right]=\mathbf{0}.$$

(60)

Then, we obtain

$$\widehat{\mathit{\gamma }}-\mathit{\gamma }={\left(\frac{1}{I}\sum _{i=1}^I{\mathit{M}}_{\mathit{\gamma }}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })\right)}^{-1}\left[\frac{1}{I}\sum _{i=1}^I\mathit{m}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })+\frac{1}{I}\sum _{i=1}^I{\mathit{M}}_{\mathit{y}}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })({\widehat{\mathit{y}}}_i-{\mathit{y}}_i)\right].$$

(61)

By definition, we have for $I\to \infty$

$$\frac{1}{I}\sum _{i=1}^I\mathit{m}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })\stackrel{p}{\to }{\mathit{S}}_0({\mathit{\gamma }}_{\infty },G)=\int \mathit{m}(\mathit{y},{\mathit{\gamma }}_{\infty })\mathrm{d}G\left(\tilde{\mathit{y}}\right)=\mathbf{0}.$$

(62)

Moreover, the following limit exists as in Section 4.1.3:

$$\frac{1}{I}\sum _{i=1}^I{\mathit{M}}_{\mathit{\gamma }}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })\stackrel{p}{\to }\mathit{A}\left({\mathit{\gamma }}_{\infty }\right)=-\int {\mathit{M}}_{\mathit{\gamma }}(\mathit{y},{\mathit{\gamma }}_{\infty })\mathrm{d}G\left(\tilde{\mathit{y}}\right).$$

(63)

Because ${\widehat{\mathit{y}}}_i-{\mathit{y}}_i\stackrel{p}{\to }\mathbf{0}$ for $P\to \infty$, the second term in the right bracket in 61 vanishes asymptotically for $I\to \infty$

$$\frac{1}{I}\sum _{i=1}^I{\mathit{M}}_{\mathit{y}}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })({\widehat{\mathit{y}}}_i-{\mathit{y}}_i)\stackrel{p}{\to }\mathbf{0}.$$

(64)

For the computation of the covariance matrix, we have

$$\frac{1}{I}\sum _{i=1}^I\mathit{m}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })\mathit{m}{({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })}^{\top }\stackrel{p}{\to }\mathit{B}\left({\mathit{\gamma }}_{\infty }\right)=\int \mathit{m}(\mathit{y},{\mathit{\gamma }}_{\infty })\mathit{m}{(\mathit{y},{\mathit{\gamma }}_{\infty })}^{\top }\mathrm{d}G\left(\tilde{\mathit{y}}\right)\mathrm{and}$$

(65)

$$\frac{1}{I}\sum _{i=1}^I{\mathit{M}}_{\mathit{y}}({\mathit{y}}_i,{\mathit{\gamma }}_{\infty }){\mathit{V}}_i\left({\mathit{y}}_i\right){\mathit{M}}_{\mathit{y}}{({\mathit{y}}_i,{\mathit{\gamma }}_{\infty })}^{\top }\stackrel{p}{\to }\mathit{C}\left({\mathit{\gamma }}_{\infty }\right)=\int {\mathit{M}}_{\mathit{y}}(\mathit{y},{\mathit{\gamma }}_{\infty }){\mathit{V}}_i\left(\mathit{y}\right){\mathit{M}}_{\mathit{y}}{(\mathit{y},{\mathit{\gamma }}_{\infty })}^{\top }\mathrm{d}G\left(\tilde{\mathit{y}}\right).$$

(66)

This shows the asymptotic normal distribution when the simultaneous inference with respect to persons and items is conducted:

$$\widehat{\mathit{\gamma }}\sim \mathrm{AN}\left({\mathit{\gamma }}_{\infty },\frac{1}{I}\mathit{A}{\left({\mathit{\gamma }}_{\infty }\right)}^{-1}\left(\mathit{B}\left({\mathit{\gamma }}_{\infty }\right)+\frac{1}{P}\mathit{C}\left({\mathit{\gamma }}_{\infty }\right)\right)\mathit{A}{\left({\mathit{\gamma }}_{\infty }\right)}^{-\top }\right).$$

(67)

It evident is from (67) that the number of persons and the number of items are part of the statistical inference. The involved matrices $\mathit{A}\left({\mathit{\gamma }}_{\infty }\right)$, $\mathit{B}\left({\mathit{\gamma }}_{\infty }\right)$, and $\mathit{C}\left({\mathit{\gamma }}_{\infty }\right)$ can be estimated from sample data. However, for example, in (65), the true identified item parameters ${\mathit{y}}_i$ in the left term has to be replaced by the estimated item parameters ${\widehat{\mathit{y}}}_i$ which can cause slight biases in estimated variance matrices. Because of this disadvantage, we propose resampling techniques for the simultaneous inference of standard errors and linking errors in the next section.

6. Resampling Methods for the Simultaneous Assessment of Standard Errors and Linking Errors

We now derive estimation formulas for resampling methods [102,103] for persons and items. The derivation is motivated by assuming the following data-generating model

$$Y_{pi}=\mu +u_p+v_i+e_{pi},\mathrm{Var}\left(u_p\right)={\sigma }_{\mathcal{P}}^2,\mathrm{Var}\left(v_i\right)={\sigma }_{\mathcal{I}}^2,\mathrm{Var}\left(e_{pi}\right)={\sigma }_{\mathcal{P}\times \mathcal{I}}^2.$$

(68)

where $Y_{pi}$ is the observed data for person p (or person groups) and item i (or item groups). The random variables $u_p$, $v_i$, and $e_{pi}$ are all independent of each other. We now derive the variance for the mean estimate $\widehat{\mu }$:

$$\widehat{\mu }=\frac{1}{PI}\sum _{p=1}^P\sum _{i=1}^I{X}_{pi}.$$

(69)

Its variance is given by

$$\mathrm{Var}\left(\widehat{\mu }\right)=\frac{{\sigma }_{\mathcal{P}}^2}{P}+\frac{{\sigma }_{\mathcal{I}}^2}{I}+\frac{{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}{PI}.$$

(70)

The variance in (70) contains error sources for persons and items. Hence, it allows a simultaneous inference for both error facets. Following the terminology of errors in item response modeling for the large-scale assessment of students [93], the variance ${\sigma }_{\mathcal{P}}^2/P$ quantifies sampling error due to sampling persons, the variance ${\sigma }_{\mathcal{I}}^2/I$ linking error due to sampling items, and the variance ${\sigma }_{\mathcal{P}\times \mathcal{I}}^2/PI$ can be interpreted as measurement error.

6.1. Single Jackknife (SJK)

The classical single jackknife (SJK; [104,105,106,107]) approach removes one unit (e.g., a (group of) person(s) or a (group of) item(s)) from an analysis for computing standard errors. First, we investigate the jackknife estimate in which only persons are removed. Let ${\widehat{\mu }}_{(-p,0)}$ be the mean estimate in which person p is removed:

$${\widehat{\mu }}_{(-p,-0)}=\frac{1}{(P-1)I}\sum _{\genfrac{}{}{0pt}{}{q=1}{q\ne p}}^P\sum _{i=1}^I{X}_{qi}$$

(71)

$${\widehat{\mu }}_{(-p,-0)}-\widehat{\mu }=-\frac{1}{P}{v}_p+\sum _{q\ne p}\frac{1}{P(P-1)}{v}_p-\frac{1}{PI}\sum _i{e}_{pi}+\sum _{q\ne p}\sum _i\frac{1}{(P-1)IP}{e}_{qi}\left[3ex\right]$$

(72)

We now derive the expected value of the square in Equation (72):

$$\mathrm{E}{({\widehat{\mu }}_{(-p,-0)}-\widehat{\mu })}^2=\frac{1}{P(P-1)}{\sigma }_{\mathcal{P}}^2+\frac{1}{P(P-1)I}{\sigma }_{\mathcal{P}\times \mathcal{I}}^2.$$

(73)

Now, we define

$$S_{\mathcal{P}}^2=\sum _{p=1}^P{({\widehat{\mu }}_{(-p,-0)}-\widehat{\mu })}^2.$$

(74)

By using (72), we now obtain

$$\mathrm{E}{S}_{\mathcal{P}}^2={\displaystyle \frac{1}{P-1}{\sigma }_{\mathcal{P}}^2+\frac{1}{(P-1)I}{\sigma }_{\mathcal{P}\times \mathcal{I}}^2.}$$

(75)

Equation (75) allows the computation of the standard error associated with person sampling. From Equation (70), we can attribute the variance ${\sigma }_{\mathcal{P}\times \mathcal{I}}^2/P$ to person sampling. From (75), we get by replacing the expected value $\mathrm{E}{S}_{\mathcal{P}}^2$ with the observed value $S_{\mathcal{P}}^2$

$${\displaystyle \frac{1}{P}{\sigma }_{\mathcal{P}}^2=\frac{P-1}{P}{S}_{\mathcal{P}}^2-\frac{1}{PI}{\sigma }_{\mathcal{P}\times \mathcal{I}}^2.}$$

(76)

In the single jackknife, the person-by-item interaction variance component ${\sigma }_{\mathcal{P}\times \mathcal{I}}^2$ is typically ignored and the variance due to person sampling is, hence, estimated by $\frac{P-1}{P}{S}_{\mathcal{P}}^2$.

Similarly, we can derive the properties of the SJK estimate in which a single item i (or an item group) is removed from the analysis:

$${\widehat{\mu }}_{(-0,-i)}=\frac{1}{P(I-1))}\sum _{p=1}^P\sum _{\genfrac{}{}{0pt}{}{j=1}{j\ne i}}^I{X}_{pj}.$$

(77)

The SJK variance estimate for item sampling utilizes the sum of squares term

$$S_{\mathcal{I}}^2=\sum _{i=1}^I{({\widehat{\mu }}_{(-0,-i)}-\widehat{\mu })}^2.$$

(78)

By changing the indices i and p in Equation (75), we obtain

$$\mathrm{E}{S}_{\mathcal{I}}^2=\frac{1}{I-1}{\sigma }_{\mathcal{I}}^2+\frac{1}{P(I-1)}{\sigma }_{\mathcal{P}\times \mathcal{I}}^2.$$

(79)

By replacing the expected value $\mathrm{E}{S}_{\mathcal{I}}^2$ with the observed value $S_{\mathcal{I}}^2$, the quantity $\frac{I-1}{I}{S}_{\mathcal{I}}^2$ is used as the variance estimate concerning the item facet. For the joint inference of persons and items, the variance terms for persons and items are added:

$$\widehat{\mathrm{Var}}\left(\widehat{\mu }\right)=\frac{P-1}{P}{S}_{\mathcal{P}}^2+\frac{I-1}{I}{S}_{\mathcal{I}}^2.$$

(80)

Note that this variance estimate is biased because

$$\mathrm{E}\left(\widehat{\mathrm{Var}}\left(\widehat{\mu }\right)\right)-\mathrm{Var}\left(\widehat{\mu }\right)=\frac{1}{PI}{\sigma }_{\mathcal{P}\times \mathcal{I}}^2>0.$$

(81)

Consequently, so-called double jackknife resampling should be employed to remove the bias from the estimated variance.

6.2. Double Jackknife (DJK)

The double jackknife (DJK; [104,106,108]) removes a person (or a group of persons) and an item (or a group of items) from an analysis for the determination of the standard error. The elimination and repeated analysis is carried out for all persons and items. Let ${\widehat{\mu }}_{(-p,-i)}$ be the mean estimate in which the person p and item i are removed. In more detail, it is

$${\widehat{\mu }}_{(-p,-i)}=\frac{1}{(P-1)(I-1)}\sum _{\genfrac{}{}{0pt}{}{q=1}{q\ne p}}^P\sum _{\genfrac{}{}{0pt}{}{j=1}{j\ne i}}^I{X}_{qj}.$$

(82)

The estimate ${\widehat{\mu }}_{(-p,-0)}$ only removes person p, and the estimate ${\widehat{\mu }}_{(-0,-i)}$ only removes item i. The corresponding estimates have already been studied as SJK estimates in Section 6.1.

We now consider an analysis in which one person and one item are removed. One obtains

$$\begin{array}{ccc}\hfill {\widehat{\mu }}_{(-p,-i)}-\widehat{\mu }& =& {\displaystyle -\frac{1}{P}{v}_p+\sum _{q\ne p}\frac{1}{P(P-1)}{v}_q-\frac{1}{I}{u}_i+\sum _{j\ne i}\frac{1}{I(I-1)}{u}_j}\hfill \\ \hfill & & {\displaystyle -\frac{1}{PI}\left(e_{pi}+\sum _{j\ne i}{e}_{pj}+\sum _{q\ne p}{e}_{qi}\right)+\sum _{q\ne p}\sum _{j\ne i}\frac{P+I-1}{(P-1)(I-1)PI}{e}_{qj}}\hfill \end{array}$$

(83)

It follows that

$${\displaystyle \mathrm{E}{({\widehat{\mu }}_{(-p,-i)}-\widehat{\mu })}^2=\frac{1}{P(P-1)}{\sigma }_{\mathcal{P}}^2+\frac{1}{I(I-1)}{\sigma }_{\mathcal{I}}^2+\frac{P+I-1}{PI(P-1)(I-1)}{\sigma }_{\mathcal{P}\times \mathcal{I}}^2.}$$

(84)

Now, define

$$S_{\mathcal{P}\times \mathcal{I}}^2=\sum _{p=1}^P\sum _{i=1}^I{({\widehat{\mu }}_{(-p,-i)}-\widehat{\mu })}^2.$$

(85)

We then obtain by using (84)

$${\displaystyle \mathrm{E}{S}_{\mathcal{P}\times \mathcal{I}}^2=\frac{I}{P-1}{\sigma }_{\mathcal{P}}^2+\frac{P}{I-1}{\sigma }_{\mathcal{I}}^2+\frac{P+I-1}{(P-1)(I-1)}{\sigma }_{\mathcal{P}\times \mathcal{I}}^2.}$$

(86)

One can use Equations (75), (79) and (86) as estimating equations by equating the expected values of the sum of squares by their observed counterparts. We have three equations for three unknowns

$$\begin{array}{ccc}{S}_{\mathcal{P}}^2& =& {\displaystyle \frac{1}{P-1}{\sigma }_{\mathcal{P}}^2+\frac{1}{(P-1)I}{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}\hfill \\ S_{\mathcal{I}}^2& =& {\displaystyle \frac{1}{I-1}{\sigma }_{\mathcal{I}}^2+\frac{1}{(I-1)P}{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}\hfill \\ S_{\mathcal{P}\times \mathcal{I}}^2& =& {\displaystyle I\frac{1}{P-1}{\sigma }_{\mathcal{P}}^2+P\frac{1}{I-1}{\sigma }_{\mathcal{I}}^2+(P+I-1)\frac{1}{(P-1)(I-1)}{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}\hfill \end{array}$$

(87)

We further simplify (87) to

$$\begin{array}{ccc}(P-1)I{S}_{\mathcal{P}}^2& =& {\displaystyle I{\sigma }_{\mathcal{P}}^2+{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}\hfill \\ P(I-1)S_{\mathcal{I}}^2& =& {\displaystyle P{\sigma }_{\mathcal{I}}^2+{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}\hfill \\ (P-1)(I-1)S_{\mathcal{P}\times \mathcal{I}}^2& =& {\displaystyle I(I-1){\sigma }_{\mathcal{P}}^2+P(P-1){\sigma }_{\mathcal{I}}^2+(P+I-1){\sigma }_{\mathcal{P}\times \mathcal{I}}^2}\hfill \end{array}$$

(88)

Now substitute the first and second equation in (88) in the third equation. We obtain

$${\displaystyle (P-1)(I-1)S_{\mathcal{P}\times \mathcal{I}}^2=(I-1)(P-1)I{S}_{\mathcal{P}}^2+(P-1)P(I-1)S_{\mathcal{I}}^2+{\sigma }_{\mathcal{P}\times \mathcal{I}}^2.}$$

(89)

Hence, we get from (89)

$${\displaystyle \frac{{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}{PI}=\frac{(P-1)(I-1)}{PI}\left[S_{\mathcal{P}\times \mathcal{I}}^2-\frac{1}{P}{S}_{\mathcal{P}}^2-\frac{1}{I}{S}_{\mathcal{I}}^2\right].}$$

(90)

Further, the variance components for persons and items can be computed as

$$\begin{gathered} {\displaystyle \begin{array}{ccc}\hfill {\displaystyle \frac{{\sigma }_{\mathcal{P}}^2}{P}}& =& {\displaystyle \frac{P-1}{P}{S}_{\mathcal{P}}^2-\frac{{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}{PI}\mathrm{and}}\hfill \end{array} \\ \frac{{\sigma }_{\mathcal{I}}^2}{I}=\frac{I-1}{I}{S}_{\mathcal{I}}^2-\frac{{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}{PI}.} \end{gathered}$$

(91)

The quantities in (90) and (91) can be used to estimate the population variance defined in (70). The crucial issue is how to handle negative variance estimates in estimation. Based on experience from preliminary simulation studies, the following variance estimate turned out to be most satisfactory:

$$\widehat{\mathrm{Var}}\left(\widehat{\mu }\right)=\frac{P-1}{P}{S}_{\mathcal{P}}^2+\frac{I-1}{I}{S}_{\mathcal{I}}^2-\frac{{\widehat{\sigma }}_{\mathcal{P}\times \mathcal{I}}^2}{PI},$$

(92)

where ${\widehat{\sigma }}_{\mathcal{P}\times \mathcal{I}}^2=max({\sigma }_{\mathcal{P}\times \mathcal{I}}^2,0)$ is nonnegative, and ${\sigma }_{\mathcal{P}\times \mathcal{I}}^2$ is defined in Equation (90).

6.3. Single Half Sampling (SHS)

In single half sampling (SHS; [107]), half of the sample is used to reanalyze the data to compute standard errors. Let ${\widehat{\mu }}_{\mathcal{P},h}$ be the h-th half sample for persons in which half of the persons are sampled. Without loss of generality, let P be even. The h-th half sample consists of $P/2$ persons. We define half sample h in which the first $P/2$ persons are sampled and compute the mean estimate

$${\widehat{\mu }}_{\mathcal{P},h}=\frac{1}{I(P/2)}\sum _{p=1}^{P/2}\sum _{i=1}^I{X}_{pi}.$$

(93)

Then, we obtain

$${\widehat{\mu }}_{\mathcal{P},h}-\widehat{\mu }=\frac{1}{P}\sum _{p=1}^{P/2}{u}_p-\frac{1}{P}\sum _{p=P/2+1}^P{u}_p+\frac{1}{PI}\sum _{p=1}^{P/2}{e}_{pi}-\frac{1}{P}\sum _{p=P/2+1}^{PI}{e}_{pi}.$$

(94)

Hence, we get from (95)

$$\mathrm{E}{({\widehat{\mu }}_{\mathcal{P},h}-\widehat{\mu })}^2=\frac{{\sigma }_{\mathcal{P}}^2}{P}+\frac{{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}{PI}.$$

(95)

Now, there are H (potentially balanced) half samples (see [102]) with estimates ${\widehat{\mu }}_{\mathcal{P},h}$. Define the variance

$$U_{\mathcal{P}}^2=\frac{1}{H}\sum _{h=1}^H{({\widehat{\mu }}_{\mathcal{P},h}-\widehat{\mu })}^2.$$

(96)

Using (95), it follows that

$$\mathrm{E}{U}_{\mathcal{P}}^2=\frac{{\sigma }_{\mathcal{P}}^2}{P}+\frac{{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}{PI}.$$

(97)

Similarly, one can consider half samples of items. Assume that in half sample k, the first $I/2$ items are sampled. Let

$${\widehat{\mu }}_{\mathcal{I},k}=\frac{1}{P(I/2)}\sum _{p=1}^P\sum _{i=1}^{I/2}{X}_{pi}.$$

(98)

One can define the variance in estimates due to different half samples of items. Define the variance

$$U_{\mathcal{I}}^2=\frac{1}{K}\sum _{h=1}^K{({\widehat{\mu }}_{\mathcal{I},k}-\widehat{\mu })}^2.$$

(99)

Using the same derivations, we get

$$\mathrm{E}{U}_{\mathcal{I}}^2=\frac{{\sigma }_{\mathcal{I}}^2}{I}+\frac{{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}{PI}.$$

(100)

Based on the expected values in (97) and (100), one can define a variance estimate of $\widehat{\mu }$ by adding the variance components regarding persons and items as

$$\widehat{\mathrm{Var}}\left(\widehat{\mu }\right)=U_{\mathcal{P}}^2+U_{\mathcal{I}}^2.$$

(101)

Notably, this estimate is positively biased because

$$\mathrm{E}\left(\widehat{\mathrm{Var}}\left(\widehat{\mu }\right)\right)-\mathrm{Var}\left(\widehat{\mu }\right)=\frac{{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}{PI}>0.$$

(102)

As in the case of SJK, SHS also results in a biased variance estimate. In the next section, we investigate double half sampling that removes the bias component.

6.4. Double Half Sampling (DHS)

In double half sampling (DHS), half samples of persons and items are created and the analysis is replicated for these half samples. Let h be a half sample of persons, and k be a half sample of items for this dataset of persons. Let ${\widehat{\mu }}_{\mathcal{I}:\mathcal{P},kh}$ be the mean estimate for the half sample for persons and items and ${\widehat{\mu }}_{\mathcal{P},h}$ be the estimate for the half sample of persons.

Define the variance

$$U_{\mathcal{I}:\mathcal{P}}^2=\frac{1}{KH}\sum _{h=1}^H\sum _{h=1}^K{({\widehat{\mu }}_{\mathcal{I}:\mathcal{P},kh}-{\widehat{\mu }}_{\mathcal{P},h})}^2.$$

(103)

Using the same derivation as in (100), one obtains

$$\mathrm{E}{U}_{\mathcal{I}:\mathcal{P}}^2=\frac{{\sigma }_{\mathcal{I}}^2}{I}+\frac{{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}{(P/2)I}=\frac{{\sigma }_{\mathcal{I}}^2}{I}+2\frac{{\sigma }_{\mathcal{P}\times \mathcal{I}}^2}{PI}.$$

(104)

Hence, an unbiased estimate of the variance for $\widehat{\mu }$ using DHS is obtained by

$$\widehat{\mathrm{Var}}\left(\widehat{\mu }\right)=U_{\mathcal{P}}^2+U_{\mathcal{I}}^2-\frac{{\widehat{\sigma }}_{\mathcal{P}\times \mathcal{I}}^2}{PI},$$

(105)

where $\frac{{\widehat{\sigma }}_{\mathcal{P}\times \mathcal{I}}^2}{PI}=max(U_{\mathcal{I}:\mathcal{P}}^2-U_{\mathcal{P}}^2,0)$.

In practice, one can use balanced half samples based on Hadamard matrices for the most efficient variance estimates that minimize the Monte Carlo error for creating half samples [102]. In the simulation study (see Section 8), only balanced half samples are considered.

6.5. Double Bootstrap

It might be tempting to consider a double bootstrap resampling approach of persons and items as an alternative to DJK and DHS [104,109,110,111]. We believe that bootstrapping items should not be recommended because duplicating items introduces additional local dependence in IRT models, which, in turn, induces bias in estimated item parameters and linking parameters. Hence, the variability obtained from a double bootstrap will also include portions of bias.

7. Simulation Study 1: Comparing the Performance of Different Linking Methods

In Simulation Study 1, we compare the performance of robust and nonrobust linking methods for the RM in the presence and absence of random DIF. This study systematically compares several robust linking methods. In particular, the recently proposed GI method is compared with alternative methods.

7.1. Design

Data were simulated according to the RM with random DIF in two groups. In the first group, the ability distributed was simulated as $\theta \sim \mathrm{N}(0,1)$. In the second group, we simulated $\theta \sim \mathrm{N}(0.5,1)$ (i.e., ${\mu }_2=0.5$). Item difficulties $b_i$ were fixed in the simulation and were chosen equidistant in the interval $[-2,2]$. Hence, in this study, we assumed fixed item difficulties $b_i$, but simulated random DIF effects $e_i$ according to a mixture distribution $G=(1-{\pi }_{\mathrm{bias}})G_{\mathrm{AMI}}+{\pi }_{\mathrm{bias}}{G}_{\mathrm{bias}}$ (see Section 2.2). The distribution of DIF effects reference items was chosen as a centered normal distribution; that is, $G_{\mathrm{AMI}}=\mathrm{N}(0,{\tau }_{\mathrm{AMI}}^2)$. For the distribution of DIF effects of biased items $G_{\mathrm{bias}}$, we chose a two-point distribution for balanced DIF with values $-{\delta }_{\mathrm{bias}}$ and ${\delta }_{\mathrm{bias}}$ and corresponding probabilities ${\pi }_{\mathrm{bias}}/2$. For unbalanced DIF, we simulated a one-point distribution at ${\delta }_{\mathrm{bias}}$ with probability ${\pi }_{\mathrm{bias}}$ which favored the first group. In the simulation, we fixed ${\delta }_{\mathrm{bias}}$ to 0.60. The bias for MM linking is expected to be $-{\pi }_{\mathrm{bias}}{\delta }_{\mathrm{bias}}$ (see Equation (11)). It vanishes for balanced DIF and is a function of ${\pi }_{\mathrm{bias}}$ in the case of unbalanced DIF.

In the simulation, five factors were varied. First, we chose the sample size N of persons as 250, 500, 1000, and 5000. Second, we varied the number of items by $I=20$ and $I=40$. Third, we chose the proportion of biased items ${\pi }_{\mathrm{bias}}=0,0.1,0.3$. With ${\pi }_{\mathrm{bias}}=0$, no biased DIF items were simulated. Fourth, we varied the standard deviation (SD) of DIF effects of reference items ${\tau }_{\mathrm{AMI}}$ as 0, 0.1, 0.2, and 0.3. Fifth, we simulated three different distributions of DIF effects if ${\tau }_{\mathrm{AMI}}>0$: a normal distribution, a uniform distribution, and a t-distribution with four degrees of freedom. With ${\tau }_{\mathrm{AMI}}=0$, reference items do not have DIF effects. The distributions of DIF effects were appropriately scaled in order to match the SD ${\tau }_{\mathrm{AMI}}$. In total, 1000 datasets were simulated and analyzed in each condition.

7.2. Analysis

The RM model was separately estimated in the two groups. The linking methods introduced in Section 3 were applied. We chose a cutoff value of 0.4 for DIF detection in the EL method. In ATR linking, we chose trimming proportions of 0.20 and 0.40. In BSQ linking, we chose 0.4 as the threshold parameter K. IA was estimated using the powers $p=1,0.5,0.25$, and 0.1. GI linking was utilized with powers 1 and 2. HAE linking was specified with powers $p=2$, 1, 0.5, 0.25, and 0.1.

The parameter of interest was the estimated group mean difference ${\widehat{\mu }}_2$. For this parameter, the bias and root mean square error (RMSE) were computed. To reduce the dependence of the RMSE from the sample size and the number of items, we computed a relative RMSE for which the RMSE of a linking method is divided by the RMSE of the linking method with the best performance. Hence, this relative RMSE possesses the lowest value of 100 for the best linking method.

To summarize the contribution of each of the manipulated factors in the simulation, we conducted an analysis of variance (ANOVA). We used a variance decomposition for assessing the importance in the presence and absence of DIF.

Moreover, we classified linking methods on whether they showed satisfactory performance in a particular condition. We defined satisfactory performance for the bias if the absolute bias in the estimated mean ${\widehat{\mu }}_2$ was smaller than 0.01. An estimator had satisfactory performance concerning the RMSE if the relative RMSE was smaller than 125.

In all analyses, the statistical software R [112] was used. The R package sirt [113] was employed for estimating the RM model with marginal maximum likelihood as the estimation method. The linking methods were estimated using R functions particularly written for this paper.

7.3. Results

In Table 1, the variance decomposition of the ANOVA summarized across conditions of no DIF is presented. For bias, sample size N, the number of items I as well as linking methods (Meth in Table 1) have an impact. However, as we will see later, the bias is of non-negligible size in the situation of no DIF. For RMSE, linking methods constitute the major source of differences. In contrast, sample size and the number of items only have small effects on the RMSE.

Table 1. Variance proportions of different factors in the simulation study for bias and RMSE in the condition of no differential item functioning (DIF).

Source	Bias	RMSE
N	31.9	0.6
I	2.3	1.4
Meth	23.1	95.0
N × I	20.2	0.1
N × Meth	14.7	2.2
I × Meth	1.3	0.6
Residual	6.6	0.2

Note.N = sample size; I = number of items; Meth = linking method; Percentage values larger than 1.0 are printed in bold.

In Table 2, the variance decomposition of the ANOVA summarized across conditions of balanced and unbalanced DIF, respectively, is presented. All terms up to three-way interactions were included. For balanced DIF (column BAL), RMSE is more important than bias. It is evident that linking methods produced the largest variability in estimates, followed by the SD ${\tau }_{\mathrm{AMI}}$ of DIF effects of reference items, the proportion of biased items ${\pi }_{\mathrm{bias}}$, sample size N, and the type of distribution (column Dist) of DIF effects. For unbalanced DIF, the bias is primarily affected by ${\pi }_{\mathrm{bias}}$ and ${\tau }_{\mathrm{AMI}}$ and their interaction. Like for balanced DIF, the linking method substantially explains the variability in the RMSE of group mean differences.

Table 2. Variance proportions of different factors in the simulation study for bias and RMSE for balanced and unbalanced differential item functioning (DIF).

Source	BAL		UNBAL
	Bias	RMSE	Bias	RMSE
N	0.3	5.5	1.1	9.8
I	0.2	0.1	0.0	0.2
Meth	29.7	39.5	7.5	16.5
${\pi }_{\mathrm{bias}}$	0.2	5.7	40.2	1.9
${\tau }_{\mathrm{AMI}}$	6.9	9.2	20.6	0.7
Dist	3.4	4.8	2.1	0.4
N ×  I	0.0	0.1	0.0	0.3
N ×  Meth	1.0	4.5	0.8	8.1
N ×  ${\pi }_{\mathrm{bias}}$	0.1	1.2	0.5	2.5
N ×  ${\tau }_{\mathrm{AMI}}$	0.7	1.0	0.3	2.3
N ×  Dist	0.6	1.9	0.2	0.2
I ×  Meth	0.7	0.2	0.1	0.3
I ×  ${\pi }_{\mathrm{bias}}$	0.0	0.0	0.0	0.3
I ×  ${\tau }_{\mathrm{AMI}}$	0.1	0.1	0.1	0.1
I ×  Dist	0.0	0.0	0.0	0.0
Meth ×  ${\pi }_{\mathrm{bias}}$	3.1	5.2	1.4	5.3
Meth ×  ${\tau }_{\mathrm{AMI}}$	12.8	5.4	5.4	13.9
Meth ×  Dist	5.6	1.8	0.4	0.8
${\pi }_{\mathrm{bias}}$ ×  ${\tau }_{\mathrm{AMI}}$	0.2	0.9	10.1	3.1
${\pi }_{\mathrm{bias}}$ ×  Dist	0.6	0.0	0.5	0.7
${\tau }_{\mathrm{AMI}}$ ×  Dist	2.1	1.9	1.7	1.2
N ×  I ×  Meth	0.3	0.1	0.0	0.2
N ×  I ×  ${\pi }_{\mathrm{bias}}$	0.0	0.0	0.0	0.1
N ×  I ×  ${\tau }_{\mathrm{AMI}}$	0.1	0.1	0.0	0.1
N ×  I ×  Dist	0.0	0.0	0.0	0.0
N ×  Meth ×  ${\pi }_{\mathrm{bias}}$	1.0	3.2	0.4	4.8
N ×  Meth ×  ${\tau }_{\mathrm{AMI}}$	2.9	1.3	0.4	9.7
N ×  Meth ×  Dist	1.3	0.6	0.0	0.4
N ×  ${\pi }_{\mathrm{bias}}$ ×  ${\tau }_{\mathrm{AMI}}$	0.2	0.1	0.3	2.0
N ×  ${\pi }_{\mathrm{bias}}$ ×  Dist	0.2	0.0	0.0	0.5
N ×  ${\tau }_{\mathrm{AMI}}$ ×  Dist	0.5	0.4	0.1	0.4
I ×  Meth ×  ${\pi }_{\mathrm{bias}}$	0.3	0.0	0.0	0.1
I ×  Meth ×  ${\tau }_{\mathrm{AMI}}$	0.9	0.4	0.1	0.3
I ×  Meth ×  Dist	0.4	0.1	0.0	0.0
I ×  ${\pi }_{\mathrm{bias}}$ ×  ${\tau }_{\mathrm{AMI}}$	0.0	0.0	0.1	0.0
I ×  ${\pi }_{\mathrm{bias}}$ ×  Dist	0.0	0.0	0.0	0.0
I ×  ${\tau }_{\mathrm{AMI}}$ ×  Dist	0.1	0.0	0.0	0.0
Meth ×  ${\pi }_{\mathrm{bias}}$ ×  ${\tau }_{\mathrm{AMI}}$	2.8	0.9	3.5	4.6
Meth ×  ${\pi }_{\mathrm{bias}}$ ×  Dist	3.3	0.2	0.2	0.4
Meth ×  ${\tau }_{\mathrm{AMI}}$ ×  Dist	4.7	0.8	0.5	0.3
${\pi }_{\mathrm{bias}}$ ×  ${\tau }_{\mathrm{AMI}}$ ×  Dist	0.6	0.0	0.4	0.2
Residual	12.2	2.6	1.0	7.7

Note. BAL = balanced DIF; UNBAL = unbalanced DIF; N = sample size; I = number of items; Meth = linking method; Dist = distribution of DIF effects; ${\pi }_{\mathrm{bias}}$ = proportion of biased DIF items; ${\tau }_{\mathrm{AMI}}$ = standard deviation of DIF effects of reference items. Percentage values larger than 1.0 are printed in bold.

Table 3 summarizes the performance of the different linking methods across all conditions with no DIF, balanced DIF, and unbalanced DIF. In the absence of DIF, all linking methods produced unbiased estimates. However, IA with small powers p of 0.25 and 0.1 as well as HAE with $p=0.1$ resulted in less precise estimates. Interestingly, GI linking always resulted in a substantially increased variability in estimated group mean differences compared to all other linking methods.

Table 3. Summary of satisfactory performance of linking methods for bias and RMSE for no, balanced and unbalanced differential item functioning (DIF).

Method	Bias			RMSE
	NODIF	BAL	UNBAL	NODIF	BAL	UNBAL
MM	100	100	0	100	100	57
EL(0.4)	100	100	58	100	68	87
ATR(0.2)	100	100	46	100	56	71
ATR(0.4)	100	100	58	100	57	85
BSQ(0.4)	100	100	72	100	35	69
IA(1)	100	100	45	100	75	73
IA(0.5)	100	100	61	100	33	59
IA(0.25)	100	100	69	88	13	35
IA(0.1)	100	100	70	75	5	14
GI(1)	100	95	63	0	0	1
GI(2)	100	96	61	0	3	18
HAE(2)	100	100	0	100	100	58
HAE(1)	100	100	46	100	69	72
HAE(0.5)	100	100	54	100	35	54
HAE(0.25)	100	100	49	100	12	24
HAE(0.1)	100	99	41	63	2	4

Note. NODIF = no DIF; BAL = balanced DIF; UNBAL = unbalanced DIF; MM = mean-mean linking; EL = elimination of DIF items with subsequent mean-mean linking; ATR = asymmetrically trimmed mean linking; BSQ = bisquare kernel linking; IA = invariance alignment; GI = Gini linking; HAE = Haebara linking; Percentage values larger than 67 are printed in bold.

In the conditions with balanced DIF (column “BAL”), all linking methods (except for GI in a few conditions) produced unbiased estimates. However, using robust linking methods (i.e., EL, ATR, BSQ, IA, GI, HAE(p) with $p\le 1$) resulted in an efficiency loss in the RMSE compared to nonrobust linking methods (i.e., MM, HAE(2)). Among the robust linking methods, MM linking with the elimination of DIF items (i.e., EL) as well as IA and HAE with $p=1$ performed best.

Finally, the situation of unbalanced DIF (column “UNBAL”) is most challenging because linking methods have to handle the presence of biased items. Notably, robust linking methods are preferred over nonrobust linking in such a situation. In particular, MM and HAE(2) always resulted in biased estimates. Among the robust linking methods, BSQ and IA with $p=0.25$ and 0.1 resulted in the least simulation conditions with biased estimates. Concerning RMSE, EL and ATR with a trimming proportion of 0.4 performed best, followed by IA with $p=1$, HAE with $p=1$, ATR with a trimming proportion of 0.2 and BSQ linking.

Table 4 shows the RMSE for balanced DIF for $I=40$ items as a function of sample sizes (N), proportion of biased items (${\pi }_{\mathrm{bias}}$), and standard deviation of DIF effects of reference items (${\tau }_{\mathrm{AMI}}$). For balanced DIF, all linking methods produced unbiased estimates (not shown in the table). However, there were slight differences between the linking methods with respect to the RMSE. In the situation of partial invariance (i.e., ${\tau }_{\mathrm{AMI}}=0$), the efficiency loss of robust linking methods compared to nonrobust linking methods MM and HAE(2) was acceptable. However, GI resulted in larger variable estimates. Moreover, note that GI linking with $p=2$ outperformed GI with $p=1$ in most conditions. Robust linking methods IA and HAE with very small power values p (e.g., $p=0.25$ or 0.1) also caused a non-negligible RMSE increase.

Table 4. RMSE for balanced differential item functioning (DIF) for $I=40$ items as a function of sample sizes (N), proportion of biased items (${\pi }_{\mathrm{bias}}$), and standard deviation of DIF effects of reference items (${\tau }_{\mathrm{AMI}}$).

${\mathit{\tau }}_{\mathbf{AMI}}$	0						0.2
${\mathit{\pi }}_{\mathbf{bias}}$	0		0.1		0.3		0		0.1		0.3
N	250	1000	250	1000	250	1000	250	1000	250	1000	250	1000
Method
MM	100	100	100	100	100	100	100	100	100	100	100	100
EL(0.4)	102	100	104	102	109	108	108	110	110	115	119	135
ATR(0.2)	105	106	104	105	109	129	108	116	108	116	119	160
ATR(0.4)	107	107	106	108	109	108	111	120	111	124	119	133
BSQ(0.4)	110	101	112	102	126	105	132	133	136	141	164	171
IA(1)	103	104	103	104	106	108	105	110	106	113	111	125
IA(0.5)	109	108	111	111	121	116	118	129	121	133	137	160
IA(0.25)	115	112	118	113	133	118	129	141	136	151	160	188
IA(0.1)	121	111	126	115	147	122	142	155	151	168	182	215
GI(1)	173	153	157	127	181	238	213	254	214	236	282	396
GI(2)	157	154	133	112	162	151	188	232	186	207	251	379
HAE(2)	100	100	100	100	100	102	100	100	100	101	101	102
HAE(1)	103	104	104	105	107	109	106	112	107	116	113	130
HAE(0.5)	110	112	111	114	117	118	115	127	118	133	135	158
HAE(0.25)	114	115	116	119	128	125	127	143	130	148	154	182
HAE(0.1)	118	118	123	125	137	135	137	153	141	164	172	207

Note. MM = mean-mean linking; EL = elimination of DIF items with subsequent mean-mean linking; ATR = asymmetrically trimmed mean linking; BSQ = bisquare kernel linking; IA = invariance alignment; GI = Gini linking; HAE = Haebara linking; RMSE values smaller than 125 are printed in bold.

The efficiency loss of robust linking methods is much larger if the reference items also possess DIF (i.e., ${\tau }_{\mathrm{AMI}}>0$). Only IA with $p=1$ can somehow compete with MM and HAE(2) linking. The variance increase in robust linking methods IA and HAE with very small powers is apparent. It also has to be stated that GI linking produced large RMSE values in balanced DIF conditions.

Table 5 shows the bias and the RMSE for unbalanced DIF for $I=40$ items as a function of sample sizes (N), proportion of biased items (${\pi }_{\mathrm{bias}}$), and standard deviation of DIF effects of reference items (${\tau }_{\mathrm{AMI}}$). All linking methods show biases in at least one condition. Notably, nonrobust linking methods MM and HAE(2) showed the largest bias. Robust linking methods reduce the bias in all conditions. The most critical condition is ${\pi }_{\mathrm{bias}}=0.3$ and ${\tau }_{\mathrm{AMI}}=0.2$. In this condition, BSQ linking has the least bias, followed by IA with small powers 0.25 and 0.1. In this condition, it is also interesting to note that biases for a large sample size of $N=1000$ are smaller than for $N=250$.

Table 5. Bias and RMSE for unbalanced differential item functioning (DIF) for $I=40$ items as a function of sample sizes (N), proportion of biased items (${\pi }_{\mathrm{bias}}$), and standard deviation of DIF effects of reference items (${\tau }_{\mathrm{AMI}}$).

	Bias								RMSE
${\mathit{\tau }}_{\mathbf{AMI}}$	0				0.2				0				0.2
${\mathit{\pi }}_{\mathbf{bias}}$	0.1		0.3		0.1		0.3		0.1		0.3		0.1		0.3
N	250	1000	250	1000	250	1000	250	1000	250	1000	250	1000	250	1000	250	1000
Method
MM	−0.06	−0.06	−0.18	−0.18	−0.06	−0.06	−0.18	−0.18	109	157	154	382	100	121	107	163
EL(0.4)	−0.02	$\phantom{-}$0.00	−0.08	−0.02	−0.03	−0.01	−0.14	−0.10	100	100	104	115	101	100	100	114
ATR(0.2)	−0.02	−0.01	−0.10	−0.06	−0.03	−0.01	−0.15	−0.12	101	103	108	161	100	101	101	127
ATR(0.4)	−0.02	−0.01	−0.08	−0.02	−0.03	−0.02	−0.14	−0.09	104	108	105	112	103	110	100	106
BSQ(0.4)	$\phantom{-}$0.00	$\phantom{-}$0.00	−0.03	$\phantom{-}$0.00	−0.02	$\phantom{-}$0.00	−0.11	−0.02	108	100	100	100	123	120	118	100
IA(1)	−0.03	−0.02	−0.12	−0.06	−0.04	−0.03	−0.16	−0.13	102	108	119	169	100	108	102	129
IA(0.5)	−0.02	−0.01	−0.08	−0.03	−0.03	−0.02	−0.13	−0.09	109	113	109	131	109	119	105	112
IA(0.25)	−0.01	−0.01	−0.06	−0.02	−0.02	−0.02	−0.12	−0.06	117	117	111	134	123	136	113	118
IA(0.1)	−0.01	−0.01	−0.05	−0.02	−0.01	−0.01	−0.12	−0.06	127	126	116	140	136	153	120	132
GI(1)	$\phantom{-}$0.03	$\phantom{-}$0.02	$\phantom{-}$0.01	$\phantom{-}$0.01	−0.01	$\phantom{-}$0.03	−0.11	−0.11	150	132	133	173	194	213	162	255
GI(2)	$\phantom{-}$0.04	$\phantom{-}$0.02	$\phantom{-}$0.00	$\phantom{-}$0.01	$\phantom{-}$0.00	$\phantom{-}$0.05	−0.13	−0.13	128	110	117	102	167	175	153	251
HAE(2)	−0.06	−0.06	−0.18	−0.18	−0.06	−0.06	−0.18	−0.18	109	156	154	380	101	121	108	163
HAE(1)	−0.03	−0.02	−0.12	−0.06	−0.04	−0.03	−0.15	−0.13	103	108	118	164	101	108	102	128
HAE(0.5)	−0.02	−0.01	−0.08	−0.03	−0.03	−0.02	−0.15	−0.10	108	113	111	132	109	119	108	124
HAE(0.25)	−0.02	−0.01	−0.08	−0.04	−0.03	−0.02	−0.16	−0.12	113	118	120	232	120	133	122	159
HAE(0.1)	−0.02	−0.01	−0.08	−0.06	−0.03	−0.02	−0.17	−0.14	119	123	132	342	130	148	132	195

Note. MM = mean-mean linking; EL = elimination of DIF items with subsequent mean-mean linking; ATR = asymmetrically trimmed mean linking; BSQ = bisquare kernel linking; IA = invariance alignment; GI = Gini linking; HAE = Haebara linking; Absolute bias values smaller than 0.05 and RMSE values smaller than 125 are printed in bold.

With respect to the RMSE, EL, ATR, BSQ, and IA with powers 0.5 and 0.25 can be recommended. It is important to emphasize that GI linking with $p=2$ performed well in the case of partial invariance (i.e., ${\tau }_{\mathrm{AMI}}=0$), but outperformed the recently proposed GI linking using $p=1$. Interestingly, DIF detection with subsequent MM linking (method EL) was also relatively effective as long as the proportion of biased items is not too large.

8. Simulation Study 2: Performance of Resampling Methods for Computing Standard Errors and Linking Errors

In Simulation Study 2, we investigate the performance of resampling methods for estimating the variability of group mean differences. DJK and DHS have not yet been systematically studied for linking methods in the literature. In particular, there is a lack of research for studying resampling methods for robust linking methods.

8.1. Design

The data generating model closely follows that of Simulation Study 1 (see Section 7.1). Only a selected number of conditions was simulated because resampling methods are computationally demanding. In contrast to Simulation Study 1, we set ${\delta }_{\mathrm{bias}}=0.3$. Only balanced DIF was simulated because the assessment of variability (and not bias) was the focus of this simulation. The proportion of biased items were chosen as ${\pi }_{\mathrm{bias}}=0$ or ${\pi }_{\mathrm{bias}}=0.3$. The SD of DIF effects for reference items ${\tau }_{\mathrm{AMI}}$ was set to 0.3. We considered sample sizes $N=500$ and $N=2000$ and fixed the number of items to $I=40$. In total, 2000 replications were conducted in each condition of the simulation study.

8.2. Analysis

To further reduce computation time, we only chose a selected number of linking methods that provided unbiased estimates in Simulation Study 1; that is, MM, ATR, IA, and HAE. We assessed the variability in estimated group mean differences with the resampling methods SJK (Equation (80)), DJK (Equation (92)), SHS (Equation (101)), and DHS (Equation (105)). We applied the resampling methods with 20 replication zones (containing 500/20 = 25 or 2000/20 = 100 persons and 40/20=2 items in each zone). Approximate balanced half sampling was used by specifying zones so that it was constructed from the upper part of a Hadamard matrix with a minimum dimension larger than 20. We computed confidence intervals based on the estimated standard errors $\widehat{s}$ by the respective methods as $\mathrm{CI}\left({\widehat{\mu }}_2\right)={\widehat{\mu }}_2\mp 1.96·\widehat{s}$. The proportion of replications in which the true difference ${\mu }_2$ is contained in $\mathrm{CI}\left({\widehat{\mu }}_2\right)$ is defined as the coverage rate. Coverage rates are classified as satisfactory if they range within the interval $[92.5,97.5]$ for a condition in a simulation. As in Simulation Study 1, we used R [112] and the R package sirt [113].

8.3. Results

In Table 6, coverage rates for the resampling methods are displayed. By construction, single resampling methods (SJK and SHS) result in slightly wider confidence intervals than double resampling methods (DJK and DHS), which, in turn, produce higher coverage rates. It can be seen that SJK and DJK failed to produce acceptable coverage rates. In particular, jackknife error estimates performed worse for robust linking methods. This is in line with results in robust statistics that JK does not work for nondifferentiable statistics. However, SJK can be used for the nonrobust linking method HAE(2). In contrast, half sampling resampling methods outperformed jackknife. As expected, SHS produced a slight overcoverage, but DHS produced acceptable coverage in all conditions. Particularly noteworthy is the fact that DHS also successfully performed for robust linking methods. Overall, these findings indicate that half sampling methods should be preferred over jackknife resampling.

Table 6. Coverage rates for linking methods for balanced differential item function for $I=40$ items as a function of sample size (N) and the proportion of biased items (${\pi }_{\mathrm{bias}}$).

Method	$\mathit{N}=500$				$\mathit{N}=2000$
	SJK	DJK	SHS	DHS	SJK	DJK	SHS	DHS
	No biased items (${\pi }_{\mathrm{bias}}=0$)
MM	94.8	92.2	95.1	93.6	93.9	91.8	94.4	93.6
ATR(0.2)	97.6	95.5	96.5	93.5	95.0	84.0	95.3	93.4
ATR(0.4)	98.9	97.4	96.8	93.7	95.9	89.6	95.4	92.7
IA(1)	98.2	96.5	96.5	94.3	93.7	89.3	94.9	93.1
IA(0.5)	99.8	99.6	97.7	94.2	98.2	97.3	97.2	93.9
IA(0.25)	99.8	99.7	98.8	95.1	99.6	99.1	98.5	94.0
HAE(2)	94.9	91.8	95.1	93.9	93.9	91.3	94.1	93.5
HAE(1)	98.5	97.9	96.9	94.3	95.2	91.9	95.3	92.8
HAE(0.5)	99.8	99.8	98.2	95.9	99.2	98.2	97.7	94.5
HAE(0.25)	99.8	99.8	98.9	95.5	99.8	99.1	98.0	93.6
	Biased items (${\pi }_{\mathrm{bias}}=0.3$)
MM	95.8	93.1	95.5	94.6	97.2	96.1	97.1	96.5
ATR(0.2)	97.8	95.3	97.1	94.2	93.6	83.8	97.1	95.9
ATR(0.4)	98.2	95.8	96.9	93.4	91.9	83.3	95.5	93.9
IA(1)	97.8	95.8	96.5	94.2	93.4	87.7	96.8	95.8
IA(0.5)	99.2	99.1	97.8	93.9	96.0	93.6	97.5	95.2
IA(0.25)	99.5	99.3	98.0	95.1	98.1	96.5	97.5	94.1
HAE(2)	95.6	93.0	96.1	94.6	97.1	96.1	97.0	96.4
HAE(1)	98.1	97.4	96.7	94.1	95.0	91.5	96.9	95.5
HAE(0.5)	99.5	99.2	98.0	95.1	97.7	95.4	98.2	96.3
HAE(0.25)	99.9	99.7	98.3	95.5	99.6	98.0	98.6	95.8

Note. SJK = single jackknife (Equation (80)); DJK = double jackknife (Equation (92)); SHS = single half sampling (Equation (101)); DHS = double half sampling (Equation (105)); MM = mean-mean linking; ATR = asymmetrically trimmed mean linking; IA = invariance alignment; HAE = Haebara linking; Coverage rates within the interval $[92.5,97.5]$ are printed in bold.

9. Discussion

In this article, we investigated the performance of robust and nonrobust linking methods as well as the assessment of standard error and linking error estimates of group mean differences. We assumed random DIF with a mixture distribution model. Items are implicitly classified into a set of reference items (that are valid for group comparisons) and biased items that potentially bias group mean differences. We studied the nonrobust linking methods mean-mean linking (MM) and Haebara linking (HAE) with $p=2$, as well as the robust linking methods based on the asymmetrically trimmed mean (ATR), elimination of DIF items with subsequent mean-mean linking (EL), bisquare linking (BSQ), invariance alignment (IA) with $p\le 1$, Haebara linking (HAE) with $p\le 1$ and Gini linking (GI).

We found that robust linking methods can be very effective in reducing biases in the presence of biased items in unbalanced DIF situations. However, in the presence of DIF on reference items (i.e., in the absence of partial invariance), robust linking methods can result in the reduced efficiency of estimates compared to nonrobust methods such as mean-mean linking or Haebara linking, in particular in the situation of balanced DIF. Our study also compared the recently proposed Gini linking with alternative linking methods. Surprisingly, GI performed worse compared to its competitors and only showed an acceptable performance using a modified GI version using a power $p=2$. In our view, it is hard to recommend a particular linking estimator in the unbalanced DIF situation. It is only evident that mean-mean linking and Haebara linking with $p=2$ are prone to bias and should not be used. Moreover, the recently proposed Gini linking produced much more variable estimates than competitive linking estimators. The usual practice in psychometrics (linking method EL) that eliminates DIF items in the first step of the analysis and computes group differences based on the DIF-free items in the second step, provides comparable results to robust linking methods (see also [47,114]). Note that we used the median as the preliminary location estimate in the EL method in the first step, which differs from the practice that employs the equal mean difficulty assumption (i.e., uses the mean instead of the median; see [52]).

We also studied the variability of group mean difference estimates due to random DIF. The randomness of DIF introduces an additional source of error (i.e., the linking error) in addition to the standard error associated with the sampling of persons. We analytically derived the distribution of the group difference through M-estimation theory. These results have importance for a (very) large number of items. Because we used a relatively small number of $I=40$ items in the simulation and large item pools are not often present in applications, we investigated (single and double) jackknife and (single and double) half sampling resampling methods for persons and items for assessing the variability in estimates of the linking methods. We found that our proposed double half sampling outperformed jackknife-based error estimates. In contrast to jackknife, half sampling can also satisfactorily be applied to nondifferentiable robust linking methods. These findings indicate that half sampling methods could find their way for assessing linking errors in empirical applications.

In this article, we focused on the estimation of group differences. In the investigation of DIF in applied research, how to choose the correct anchor is always crucial [41,50,115,116,117,118]. The studied robust linking estimators can be used to transform estimated item difficulties onto the same scale. Differences in transformed item difficulties can be investigated for DIF effects. Resampling procedures (single jackknife or single half sampling) can be employed for assessing the statistical significance of DIF effects.

As an alternative to separate scaling with subsequent robust or nonrobust linking, concurrent scaling assuming invariant item parameters can be utilized. Although such a one-step approach might be preferred from the practitioner’s point of view, the presence of DIF effects likely introduces some bias in estimated group differences [5,119]. Surprisingly, the bias is even present for balanced DIF [119]. Robust linking methods have the advantage that a few outlying DIF effects are automatically removed from group comparisons [119]. Moreover, concurrent calibration might have computational disadvantages [44,120]. As a further alternative, concurrent calibration assuming partial invariance can be pursued [47,121,122]. In this approach, DIF for items is investigated in a first step, and items that showed DIF receive group-specific item parameters in the concurrent calibration approach, while invariance is assumed for the remaining items.

Furthermore, the precision of linking estimates can be improved by including further person covariates in the analysis [123,124]. This could be particularly true if there also exist DIF effects for person covariates. There is a lack of research for studies that include person covariates in robust linking methods.

Finally, we assumed that the Rasch model was correctly specified. This assumption might be unrealistic in practice, and much more complex item response functions could have generated the item responses [23,29]. It would be interesting to study the performance of the different linking methods and the assessment of standard errors and linking errors for misspecified models. We would like to emphasize that M-estimation theory and resampling techniques also provide valid inference in the case of misspecified models. It can always be debated whether estimates from a misspecified Rasch model are practically relevant or should be interpreted. We tend to argue that parameter estimates of misspecified models summarize a population distribution, and model fitting is not always (or maybe should not be) targeted at estimating the model that has generated the data. In this sense, we think that approaches that include model error as an additional component in statistical inference [125] might be beneficial.