A Comparison of Linking Methods for Two Groups for the Two-Parameter Logistic Item Response Model in the Presence and Absence of Random Differential Item Functioning

Abstract

This article investigates the comparison of two groups based on the two-parameter logistic item response model. It is assumed that there is random differential item functioning in item difficulties and item discriminations. The group difference is estimated using separate calibration with subsequent linking, as well as concurrent calibration. The following linking methods are compared: mean-mean linking, log-mean-mean linking, invariance alignment, Haberman linking, asymmetric and symmetric Haebara linking, different recalibration linking methods, anchored item parameters, and concurrent calibration. It is analytically shown that log-mean-mean linking and mean-mean linking provide consistent estimates if random DIF effects have zero means. The performance of the linking methods was evaluated through a simulation study. It turned out that (log-)mean-mean and Haberman linking performed best, followed by symmetric Haebara linking and a newly proposed recalibration linking method. Interestingly, linking methods frequently found in applications (i.e., asymmetric Haebara linking, recalibration linking used in a variant in current large-scale assessment studies, anchored item parameters, concurrent calibration) perform worse in the presence of random differential item functioning. In line with the previous literature, differences between linking methods turned out be negligible in the absence of random differential item functioning. The different linking methods were also applied in an empirical example that performed a linking of PISA 2006 to PISA 2009 for Austrian students. This application showed that estimated trends in the means and standard deviations depended on the chosen linking method and the employed item response model.

1. Introduction

The analysis of educational and psychological tests is an important field in the social sciences. The test items (i.e., tasks presented in these tests) are often modeled using item response theory (IRT; [1,2,3]; for applications, see, e.g., [4,5,6,7,8,9,10,11,12,13,14]) models. In this article, the two-parameter logistic (2PL; [15]) IRT model is investigated to compare two groups on test items. For example, groups could be demographic groups, countries, studies, or time points. The group comparisons were carried out using linking methods [16]. A significant obstacle in applying linking methods is that the test items could behave differently in the two groups (i.e., differential item functioning), that is it cannot be expected that the two groups share a common set of statistical parameters for the test items. Such a situation is particularly important in educational large-scale assessment (LSA; [17,18,19]) studies in which several countries are compared. It can be expected that test items function differently because there are curricular differences in those countries.

The paper is structured as follows. In Section 2, the 2PL model with differential item functioning is introduced. In Section 3, several linking methods are discussed. In Section 4, we present the results of a simulation study in which these linking methods are compared. In Section 5, an empirical example using the PISA datasets from 2006 and 2009 for Austria is presented. Finally, the paper closes with a discussion in Section 6.

2. Linking Two Groups with the 2PL Model

2.1. 2PL Model

For dichotomous items $i=1,\dots ,I$, the item response function (IRF) of the 2PL model is given by [3,20]:

$$\mathrm{P}(X_i=x|\theta )=P_i(x;a_i,b_i)=\mathsf{\Psi }\left((2x-1)a_i(\theta -b_i)\right)x=0,1,$$

(1)

where $\mathsf{\Psi }\left(y\right)=exp\left(y\right)/(1+exp(y\left)\right)$ is the logistic link function, $a_i$ is the item discrimination, and $b_i$ is the item difficulty. The one-parameter logistic (1PL) IRT model (also referred to as the Rasch model; [21]) fixes all item discriminations $a_i$ to one.

The 1PL or the 2PL model is usually estimated under a local independence assumption, that is:

$$\mathrm{P}(\mathit{X}=\mathit{x})=\int \prod _{i=1}^I{P}_i(x_i,a_i,b_i)f\left(\theta \right)\mathrm{d}\theta ,$$

(2)

where f is the density of $\theta$ and $\mathit{x}=(x_1,\dots ,x_I)$. In many applications, $\theta$ is assumed to be normally distributed (i.e., $\theta \sim \mathrm{N}(\mu ,{\sigma }^2)$). In (2), the multivariate contingency table of $\mathit{X}$ with corresponding probabilities $\mathrm{P}(\mathit{X}=\mathit{x})$ involving $2^I$ item response patterns was summarized by a unidimensional latent variable $\theta$.

2.2. Linking Design

To assess the distributional differences between two groups, an appropriate linking design must be established to identify group differences. Such a linking design is displayed in Figure 1. For two groups $g=1,2$, there is a set $I_0$ of common items (also referred to as anchor items or link items) that are administered in both groups. There are also group-specific items $I_g$ that are uniquely administered in each of the two groups.

This linking design is also referred to as a common items nonequivalent group design [22]. In equating, no common items exist in many applications, but two test forms are administered to equivalent groups [23]. Equating is often performed with the goal of producing an equivalency table for the sum score in a test, while linking determines linking constants in order to identify group differences with respect to the latent variable $\theta$. In this article, we are interested in determining distributional differences between the two groups for the latent variable $\theta$.

It has to be emphasized that differences between the two groups are mainly determined by the set of common items if the linking relies on an IRT model. The employed IRT models can predict the expected performance of students on items that were not administered. The crucial assumption is that item responses on nonadministered unique items can be inferred (or imputed) from common items. This corresponds to an ignorability assumption in the missing data literature and translates into the conditional independence of item responses on unique items conditional on item responses on common items (see [24,25]). If a unidimensional IRT model with a local independence assumption (2) holds, this condition is automatically fulfilled.

2.3. Random Differential Item Functioning

Common items in the set $I_0$ are administered in two groups $g=1,2$ (see Figure 1). It is assumed that the 2PL model (see Section 2.1) holds in both groups. The situation in which item parameters do not differ between groups is called measurement invariance [26,27,28]. However, it is likely in many applications that items function differently in the two groups. The existence of group-specific item parameters is labeled differential item functioning (DIF; [29,30,31,32,33]; for applications, see, e.g., [34,35,36,37,38]). In this case, items possess group-specific item discriminations $a_{ig}$ and item difficulties $b_{ig}$. DIF in item discriminations is referred to as nonuniform DIF (NUDIF), while DIF in item difficulties is referred to as uniform DIF (UDIF) if there is no DIF in item discriminations (see [32,39,40]). Note that the local independence assumption (2) holds within each group even though the item parameters can differ across groups.

For the rest of this article, we assumed random DIF [41,42,43,44,45,46,47,48,49,50]. The main idea is that the difference of item parameters between groups is modeled by a distribution. Hence, a population perspective to a universe of items was adopted. Random DIF for item discriminations $a_{ig}$ ($i=1,\dots ,I;g=1,2$) is defined as:

$$a_{i1}=a_i-f_i\mathrm{and}{a}_{i2}=a_i+f_i,$$

(3)

where DIF effects $f_i$ are considered as a random variable that follows a distribution $F_f$. Common item discriminations $a_i$ can be considered either fixed or random. In the following, we considered them as fixed. Item difficulties $b_{ig}$ are also considered as random DIF, and DIF effects $e_i$ follow a random distribution $F_e$:

$$b_{i1}=b_i-e_i\mathrm{and}{b}_{i2}=b_i+e_i,$$

(4)

where $b_i$ are common item difficulties.

It is important to emphasize that there is an inherent indefinability of a simultaneous determination of average DIF effects and average group differences [51,52,53,54]. Hence, some structural assumptions must be posed on the distributions of DIF effects $F_e$ and $F_f$. One possible choice would be to assume $\mathrm{E}\left(e_i\right)=\mathrm{E}\left(f_i\right)=0$; that is, random DIF is is described by unsystematic differences in the item parameters (see Appendix A for further details). A frequent assumption is that DIF effects are normally distributed with zero means:

$$e_i\sim \mathrm{N}(0,{\tau }_b^2)\mathrm{and}{f}_i\sim \mathrm{N}(0,{\tau }_a^2)$$

(5)

It is impossible to assume the means of DIF effects different from zero because average DIF effects are confounded with average group differences. Note that it holds that $a_{i2}-a_{i1}=2f_i\sim \mathrm{N}(0,2{\tau }_a^2)$ and $b_{i2}-b_{i1}=2e_i\sim \mathrm{N}(0,2{\tau }_b^2)$.

As an alternative, a sparsity assumption might be posed on DIF effects. In this case, the majority of items have DIF effects of zero, while only a few items have DIF effects different from zero [44,55]. If DIF effects are considered as fixed, this situation is known as partial invariance [56,57,58]. The random DIF distribution is a mixture distribution with two classes: one class with zero effects and the other class containing DIF effects different from zero. The mixture distribution can be simultaneously estimated in an IRT model with two groups ([55]; see also [59]). Alternatively, DIF detection methods can be used to identify items whose DIF effects differ from zero [32,60,61]. These items can be removed from linking in subsequent analysis (see, e.g., [62]). However, some research has shown that simultaneous treatment of the linking and modeling of DIF effects can result in superior statistical performance [54,63,64,65,66].

We would also like to point out that DIF effects in item discriminations in Equation (3) follow an additive model. Alternatively, a multiplicative model for DIF effects can be assumed:

$$a_{i1}=a_i/f_i\mathrm{and}{a}_{i2}=a_i{f}_i,$$

(6)

which corresponds to an additive model in logarithmized item discriminations:

$$log{a}_{i1}=log{a}_i-log{f}_i\mathrm{and}log{a}_{i2}=log{a}_i+log{f}_i.$$

(7)

It is hard to decide in empirical applications whether DIF effects for item discriminations should be modeled in the untransformed metric (see Equation (3)) or a logarithmized metric (see Equation (7)).

The simulation study only considered normally distributed DIF effects with zero means and an additive model for DIF effects in item discriminations.

2.3.1. Identified Item Parameters in Separate Calibrations in the Two Groups

In Section 3, we discuss some linking methods that rely on item parameters that were obtained from separate calibrations. That is, the 2PL model was separately fitted for the two groups. For reasons of identifiability, it has to be assumed that in the first group, it holds that ${\mu }_1=0$ and ${\sigma }_1=1$. In a separate estimation for the first group with an infinite sample size, the estimated item discriminations ${\widehat{a}}_{i1}$ are equal to the data-generating parameters $a_{i1}$. The same holds for estimated item difficulties, that is ${\widehat{b}}_{i1}=b_{i1}$.

In the second group, there are group-specific item parameters $a_{i2}$ and $b_{i2}$ and distribution parameters ${\mu }_2\ne 0$ and ${\sigma }_2\ne 1$. In a separate estimation for the second group, it was assumed that the mean was set to zero, and the standard deviation (SD) was set to one. Hence, estimated item parameters also include the distribution parameters. We obtain:

$$a_{i2}(\theta -b_{i2})=a_{i2}({\sigma }_2{\theta }^{\ast }+{\mu }_2-b_{i2})=\left(a_{i2}{\sigma }_2\right)({\theta }^{\ast }-{\sigma }_2^{-1}(b_{i2}-{\mu }_2)),$$

(8)

where the standardized ability ${\theta }^{\ast }$ is standard normally distributed (i.e., $\mathrm{N}(0,1)$). From Equation (8), it follows that:

$${\widehat{a}}_{i2}=a_{i2}{\sigma }_2\mathrm{and}{\widehat{b}}_{i2}={\sigma }_2^{-1}(b_{i2}-{\mu }_2).$$

(9)

2.3.2. The Role of Normally Distributed Random DIF in Educational Assessment

It should be noted that the presence of random DIF implies that there is no single item for which the two groups share the same item parameters. Hence, all items are allowed to have different item parameters. By posing a normal distribution on DIF effects, it was assumed that DIF across groups vanishes on average for an infinite number of items. The presence of normally distributed random DIF is strongly different from the situation of partial invariance in which only a few items (or a few item parameters) possess DIF, while the rest of the items (or item parameters) do not have DIF. The two kinds of DIF effects can also simultaneously occur [65]. In our experience, we mainly observed random fluctuations of group-specific item parameters, which would correspond to normally distributed random DIF instead of to the case of partial invariance. However, it seems that ensuring partial invariance is seen as a measurement ideal in educational LSA studies [67,68,69,70]. We have argued against such a perspective elsewhere [54,65,71,72] and think that the random DIF perspective with normally distributed DIF effects is more relevant in real-world applications. Moreover, we think removing items due to DIF from group comparisons is unwise because DIF can be interpreted as construct-relevant [32,52,73,74,75]. In this situation, a group difference that relies on a purified set of items can bias estimated group differences with respect to the means and standard deviations.

3. Linking Methods

In the following, we discuss linking methods that allow estimating the distribution parameters ${\mu }_2=\mathrm{E}\left(\theta \right)$ and ${\sigma }_2=\mathrm{SD}\left(\theta \right)$ of the second group. Let ${\mathbf{\gamma }}_0$ denote item parameters $a_i$ and $b_i$ of common items $I_0$ and ${\mathbf{\gamma }}_g$ item parameters from the group-specific sets of unique items $I_g$ for $g=1,2$. Furthermore, denote by ${\mathit{X}}_g$ the matrix of observed item responses from group g for items $i\in I_0\cup I_g$. For group $g=1,2$, the log-likelihood function is defined by:

$$l(\mu ,\sigma ,{\mathbf{\gamma }}_0,{\mathbf{\gamma }}_g;{\mathit{X}}_g)=\sum _{p=1}^{N_g}log\left(\int \prod _{i\in I_0\cup I_g}{P}_i(x_{pgi};a_i,b_i)f(\theta ;\mu ,\sigma )\mathrm{d}\theta \right),$$

(10)

where $x_{pgi}$ is the item response of person p in group g at item i. The IRT model in (10) can be estimated by marginal maximum likelihood (MML) estimation and an expectation–maximization algorithm [76,77,78,79].

For reasons of identification, we define ${\mu }_1=0$ and ${\sigma }_1=1$ and identify the distribution parameters ${\mu }_2$ and ${\sigma }_2$ of the second group. Separate calibrations for the two groups can be carried out and result in group-specific item parameter estimates ${\widehat{a}}_{ig}$ and ${\widehat{b}}_{ig}$ (see Section 2.3.1). These item parameters were subsequently transformed utilizing linking methods in order to obtain estimates ${\widehat{\mu }}_2$ and ${\widehat{\sigma }}_2$. See [16,22,80,81,82] for an overview of linking methods. In the next subsection, we discuss several linking methods.

3.1. Log-Mean-Mean Linking

In log-mean-mean linking (logMM; [16]), the means of the logarithmized item discriminations and item difficulties in the two groups are set equal for identifying group means and group SDs. The SD ${\sigma }_2$ of the second group is estimated as:

$${\widehat{\sigma }}_2=exp\left(\frac{1}{|I_0|}\sum _{i\in I_0}log{\widehat{a}}_{i2}-\frac{1}{|I_0|}\sum _{i\in I_0}log{\widehat{a}}_{i1}\right),$$

(11)

where $|I_0|$ is the number of items in the set $I_0$. The estimation in (11) corresponds to the assumption of additive DIF effects in logarithmized item discriminations (see (7)). The mean ${\mu }_2$ of the second group is estimated by:

$${\widehat{\mu }}_2=-{\widehat{\sigma }}_2\frac{1}{|I_0|}\sum _{i\in I_0}{\widehat{b}}_{i2}+\frac{1}{|I_0|}\sum _{i\in I_0}{\widehat{b}}_{i1}.$$

(12)

It can be shown that logMM estimates are consistent under weak conditions. Here, consistency means that the number of common items (i.e., $|I_0|$) is tending toward infinity. Moreover, it was assumed that group-specific item parameters would have been estimated with an infinite sample size (i.e., $N\to \infty$). Consistency proofs can be easily modified to the case of finite sample sizes (use $\stackrel{p}{\to }$ as the mathematical symbol for consistency in the folllowing). Then, consistency is meant under a double sampling scheme in which the number of persons and number of items tends to infinity (i.e., $N\to \infty$ and $|I_0|\to \infty )$. We now present the consistency result.

Proposition 1. 

Assume that DIF effects $e_i$ fulfill $\mathrm{E}\left(e_i\right)=0$. For DIF effects $f_i$, one of the following conditions holds:

(i) 

For additive DIF effects $f_i$ (Equation (3)), it holds that $\mathrm{E}\left(f_i\right)=0$ and $f_i$ has a symmetric distribution;

(ii) 

For multiplicative DIF effects $f_i$ (Equation (7)), it holds that $\mathrm{E}(log{f}_i)=0$.

Then, logMM estimators ${\widehat{\mu }}_2$ and ${\widehat{\sigma }}_2$ are consistent for ${\mu }_2$ and ${\sigma }_2$, respectively:

$${\widehat{\mu }}_2\stackrel{p}{\to }{\mu }_2\mathrm{and}{\widehat{\sigma }}_2\stackrel{p}{\to }{\sigma }_2\mathrm{for}\left|I_0\right|\to \infty .$$

(13)

Proof. 

See Appendix B. □

3.2. Mean-Mean Linking

In mean-mean linking (MM; [16]), the means of untransformed item discriminations are matched. Hence, the estimation in (11) is substituted by:

$${\widehat{\sigma }}_2=\frac{{\displaystyle \frac{1}{|I_0|}\sum _{i\in I_0}{\widehat{a}}_{i2}}}{{\displaystyle \frac{1}{|I_0|}\sum _{i\in I_0}{\widehat{a}}_{i1}}}.$$

(14)

This estimation corresponds to additive DIF effects in untransformed item discriminations (see (3)). The estimation formula for ${\widehat{\mu }}_2$ in (12) remains unaltered.

It can also be shown that MM estimates are consistent under weak conditions.

Proposition 2. 

Assume that DIF effects $e_i$ fulfill $\mathrm{E}\left(e_i\right)=0$. For DIF effects $f_i$, one of the following conditions holds:

(i) 

For additive DIF effects $f_i$ (Equation (3)), it holds that $\mathrm{E}\left(f_i\right)=0$;

(ii) 

For multiplicative DIF effects $f_i$ (Equation (7)), it holds that $\mathrm{E}(log{f}_i)=0$ and $log{f}_i$ has a symmetric distribution.

Then, MM estimators ${\widehat{\mu }}_2$ and ${\widehat{\sigma }}_2$ are consistent for ${\mu }_2$ and ${\sigma }_2$, respectively:

$${\widehat{\mu }}_2\stackrel{p}{\to }{\mu }_2\mathrm{and}{\widehat{\sigma }}_2\stackrel{p}{\to }{\sigma }_2\mathrm{for}\left|I_0\right|\to \infty .$$

(15)

Proof. 

See Appendix C. □

Finally, it is instructive to study the effects on the estimates in MM linking if the true effects do not have zero means. For additive DIF effects $f_i$, we assumed $\mathrm{E}\left(f_i\right)={\delta }_f$ and $\mathrm{E}\left(e_i\right)={\delta }_e$. Using similar derivations as in Appendix C, we obtain:

$${\widehat{\sigma }}_2\stackrel{p}{\to }{\sigma }_2\frac{A+{\delta }_f}{A-{\delta }_f}={\sigma }_2\frac{1+{\delta }_f/A}{1-{\delta }_f/A},$$

(16)

where $A={lim}_{|I_0|\to \infty }\frac{1}{|I_0|}{\sum }_{i=1}^{|I_0|}{a}_i$. Setting $B={lim}_{|I_0|\to \infty }\frac{1}{|I_0|}{\sum }_{i=1}^{|I_0|}{b}_i$, we obtain:

$${\widehat{\mu }}_2\stackrel{p}{\to }{\mu }_2\frac{1+{\delta }_f/A}{1-{\delta }_f/A}-2B\frac{{\delta }_f/A}{1-{\delta }_f/A}-2{\delta }_e\frac{1}{1-{\delta }_f/A}.$$

(17)

If DIF effects do not have zero means, Equations (16) and (17) show that biased estimates for the group mean and the group standard deviation can be expected.

3.3. Haberman Linking (HAB and HAB-nolog)

Haberman linking (HAB; [71,83,84]) provides a generalization of logMM and MM to multiple groups while simultaneously estimating common item parameters $\mathit{a}$ and $\mathit{b}$. HAB consists of two steps. In the first step, group-specific SDs are estimated. In the second step, group-specific means are estimated.

The originally proposed HAB linking [83] operates on logarithmized item discriminations (HAB). To estimate ${\sigma }_2$, the linking function in HAB for the particular case of two groups is:

$$\begin{array}{ccc}\hfill H_{1,log}({\sigma }_2,\mathit{a})& =& {\displaystyle \sum _{i\in I_0\cup I_1}{\left(log{\widehat{a}}_{i1}-log{a}_i\right)}^2}\hfill \\ \hfill & & {\displaystyle +\sum _{i\in I_0\cup I_2}{\left(log{\widehat{a}}_{i2}-log{a}_i-log{\sigma }_2\right)}^2.}\hfill \\ \end{array}$$

(18)

Parameters in HAB linking are estimated as the minimizer of (18):

$$({\widehat{\sigma }}_2,\widehat{\mathit{a}})=\underset{{\sigma }_2,\mathit{a}}{arg\; min}{H}_{1,log}({\sigma }_2,\mathit{a}).$$

(19)

For $i\in I_g$ for $g=1,2$, we obtain ${\widehat{a}}_i={\widehat{a}}_{ig}$. Furthermore, for $i\in I_0$, the minimization of (18) corresponds to a two-way analysis of variance. We obtain (see Equation (A31) in Appendix D):

$${\widehat{\sigma }}_2=exp\left(\frac{1}{|I_0|}\sum _{i\in I_0}log{\widehat{a}}_{i2}-\frac{1}{|I_0|}\sum _{i\in I_0}log{\widehat{a}}_{i1}\right),$$

(20)

which shows the equivalence to logMM linking with respect to the estimation of ${\sigma }_2$.

In the second step, group means are estimated. The linking function is given as:

$${\displaystyle H_2({\mu }_2,\mathit{b})=\sum _{i\in I_0\cup I_1}{\left({\widehat{b}}_{i1}-b_i\right)}^2+\sum _{i\in I_0\cup I_2}{\left({\widehat{\sigma }}_2{\widehat{b}}_{i2}-b_i+{\mu }_2\right)}^2.}$$

(21)

The parameters are estimated as:

$$({\widehat{\mu }}_2,\widehat{\mathit{b}})=\underset{{\mu }_2,\mathit{b}}{arg\; min}{H}_2({\mu }_2,\mathit{b}).$$

(22)

Using the same derivation as for $H_{1,log}$, we obtain:

$${\widehat{\mu }}_2=-{\widehat{\sigma }}_2\frac{1}{|I_0|}\sum _{i\in I_0}{\widehat{b}}_{i2}+\frac{1}{|I_0|}\sum _{i\in I_0}{\widehat{b}}_{i1}.$$

(23)

Notably, Equation (23) coincides with the estimation in logMM linking (see Equation (12)).

As an alternative, Haberman linking can also be conducted based on untransformed item discriminations [71]. This method is labeled as HAB-nolog. It turned out that HAB-nolog outperformed HAB in some situations for multiple groups [71,84]. The linking function of HAB-nolog is given by:

$${\displaystyle H_{1,\mathrm{nolog}}({\sigma }_2,\mathit{a})=\sum _{i\in I_0\cup I_1}{\left({\widehat{a}}_{i1}-a_i-1\right)}^2+\sum _{i\in I_0\cup I_2}{\left({\widehat{a}}_{i2}-a_i-{\sigma }_2\right)}^2.}$$

(24)

Parameter estimates in HAB-nolog linking are determined by:

$$({\widehat{\sigma }}_2,\widehat{\mathit{a}})=\underset{{\sigma }_2,\mathit{a}}{arg\; min}{H}_{1,\mathrm{nolog}}({\sigma }_2,\mathit{a}).$$

(25)

The SD of the second group is given by (see Equation (A36) in Appendix D):

$${\widehat{\sigma }}_2=1+\frac{1}{|I_0|}\sum _{i\in I_0}{\widehat{a}}_{i2}-\frac{1}{|I_0|}\sum _{i\in I_0}{\widehat{a}}_{i1}.$$

(26)

The linking function for ${\mu }_2$ in HAB-nolog is the same as in HAB (see Equation (21)).

3.4. Invariance Alignment with $p=2$

Asparouhov and Muthén [85,86] proposed the method of invariance alignment (IA) to define a linking that maximizes the extent of invariant item parameters. IA can also be regarded as a linking method that handles noninvariant item parameters [71].

The IA method is based on estimated group-specific item intercepts ${\widehat{\nu }}_{ig}={\widehat{a}}_{ig}{\widehat{b}}_{ig}$ and item discriminations ${\widehat{a}}_{ig}$ obtained from separate calibrations. IA was originally formulated to detect only a few noninvariant items. It has been pointed out that IA in its original proposal is not an acceptable linking method in the presence of normally distributed DIF effects [87,88,89]. In [71], IA was studied using a general class of so-called $L_p$-type robust linking functions. The original IA formulation used $p=0.5$ [85]. For normally distributed DIF effects, $p=2$ is an adequate choice [71,89]. Hence, we investigate IA with the loss function using $p=2$ (IA2).

It has been shown that IA estimation originally formulated as a joint estimation problem can be reformulated as a two-step estimation method [71]. In the first step, group SDs are computed. In the second step, group means are computed. For two groups, ${\sigma }_2$ is estimated in the first step and ${\mu }_2$ in the second step. Note that the first group serves as the reference group (i.e., it holds that ${\mu }_1=0$ and ${\sigma }_1=1$). The formulas in [71] can be simplified to the case of two groups for providing estimates ${\widehat{\mu }}_2$ and ${\widehat{\sigma }}_2$:

$${\widehat{\sigma }}_2=\underset{{\sigma }_2}{arg\; min}\sum _{i\in I_0}{\left({\widehat{a}}_{i1}-\frac{{\widehat{a}}_{i2}}{{\sigma }_2}\right)}^2\mathrm{and}$$

(27)

$${\widehat{\mu }}_2=\underset{{\mu }_2}{arg\; min}\sum _{i\in I_0}{\left({\widehat{\nu }}_{i1}-{\widehat{\nu }}_{i2}+{\mu }_2\frac{{\widehat{a}}_{i2}}{{\widehat{\sigma }}_2}\right)}^2.$$

(28)

The estimates in (27) and (28) have close expressions (see Equations (A41) and (A45) in Appendix E).

3.5. Haebara Linking Methods (HAE-Asymm, HAE-Symm, HAE-Joint)

In contrast to the MM, logMM, HAB, HAB-nolog, and IA linking methods, Haebara (HAE) linking [90] aligns IRFs instead of directly aligning item parameters. The linking function in asymmetric HAE (HAE-asymm; [90]) linking is given as:

$$H_{\mathrm{asymm}}({\mu }_2,{\sigma }_2)=\sum _{i\in I_0}\int {\left[\mathsf{\Psi }\left({\widehat{a}}_{i1}(\theta -{\widehat{b}}_{i1})\right)-\mathsf{\Psi }\left({\sigma }_2^{-1}{\widehat{a}}_{i2}(\theta -{\sigma }_2{\widehat{b}}_{i2}-{\mu }_2)\right)\right]}^2\omega \left(\theta \right)\mathrm{d}\theta .$$

(29)

Estimated distribution parameters are obtained by minimizing (29):

$$({\widehat{\mu }}_2,{\widehat{\sigma }}_2)=\underset{{\mu }_2,{\sigma }_2}{arg\; min}{H}_{\mathrm{asymm}}({\mu }_2,{\sigma }_2).$$

(30)

The linking function (29) aligns IRFs of the second group to those in the first group. Hence, it is asymmetric because parameters in the second group are expected to behave similarly to the first group. However, the first group could alternatively be aligned to the second group. To robustify the HAE linking, both directions of alignment are considered in symmetric Haebara linking (HAE-symm; [91,92]), which employs the linking function:

$$\begin{array}{c}{\displaystyle H_{\mathrm{symm}}({\mu }_2,{\sigma }_2)=\sum _{i\in I_0}\int {\left[\mathsf{\Psi }\left({\widehat{a}}_{i1}(\theta -{\widehat{b}}_{i1})\right)-\mathsf{\Psi }\left({\sigma }_2^{-1}{\widehat{a}}_{i2}(\theta -{\sigma }_2{\widehat{b}}_{i2}-{\mu }_2)\right)\right]}^2\omega \left(\theta \right)\mathrm{d}\theta }\hfill \\ {\displaystyle +\sum _{i\in I_0}\int {\left[\mathsf{\Psi }\left({\widehat{a}}_{i1}{\sigma }_2(\theta -{\sigma }_2^{-1}({\widehat{b}}_{i1}-{\mu }_2))\right)-\mathsf{\Psi }\left({\widehat{a}}_{i2}(\theta -{\widehat{b}}_{i2})\right)\right]}^2\omega \left(\theta \right)\mathrm{d}\theta .}\hfill \end{array}$$

(31)

In a similar vein, the estimated mean and the SD for the second group is given by and defined as:

$$({\widehat{\mu }}_2,{\widehat{\sigma }}_2)=\underset{{\mu }_2,{\sigma }_2}{arg\; min}{H}_{\mathrm{symm}}({\mu }_2,{\sigma }_2).$$

(32)

A generalization of HAE linking to the general case of multiple groups was proposed in [93]. In this joint Haebara (HAE-joint) linking approach, distribution parameters are simultaneously estimated with common item parameters. The linking function in HAE-joint is defined as [65,93,94]:

$$\begin{array}{c}{\displaystyle H_{\mathrm{joint}}({\mu }_2,{\sigma }_2,\mathit{a},\mathit{b})=\sum _{i\in I_0}\int {\left[\mathsf{\Psi }\left({\widehat{a}}_{i1}(\theta -{\widehat{b}}_{i1})\right)-\mathsf{\Psi }\left(a_i(\theta -b_i)\right)\right]}^2\omega \left(\theta \right)\mathrm{d}\theta }\hfill \\ {\displaystyle +\sum _{i\in I_0}\int {\left[\mathsf{\Psi }\left({\widehat{a}}_{i2}(\theta -{\widehat{b}}_{i2})\right)-\mathsf{\Psi }\left(a_i({\sigma }_2\theta -b_i+{\mu }_2)\right)\right]}^2\omega \left(\theta \right)\mathrm{d}\theta ,}\hfill \end{array}$$

(33)

where $\mathit{a}$ and $\mathit{b}$ denote common item parameters. Parameter estimates are given by minimizing (33):

$$({\widehat{\mu }}_2,{\widehat{\sigma }}_2,\widehat{\mathit{a}},\widehat{\mathit{b}})=\underset{{\mu }_2,{\sigma }_2,\mathit{a},\mathit{b}}{arg\; min}{H}_{\mathrm{joint}}({\mu }_2,{\sigma }_2,\mathit{a},\mathit{b}).$$

(34)

A variant of joint Haebara linking was proposed in [84]. Note that in joint Haebara linking, common IRFs are aligned to group-specific IRFs.

3.6. Recalibration Linking (RC1, RC2, and RC3)

A further linking technique is recalibration (RC) linking. This is based on item parameters obtained from separate calibrations. The core idea is that group differences in distributions can be inferred by recalibrating a study with one group using item parameters from the other group. RC methods are mainly used in LSA studies such as the Programme for International Student Assessment (PISA; [95]), Progress in International Reading Literacy Study (PIRLS; [96]), and Trends in International Mathematics and Science Study (TIMSS; [97,98]).

In PISA, RC linking was employed until PISA 2012 for the 1PL model to handle model misspecifications (i.e., the data-generating IRT model deviates from the 1PL model) in linking that could artificially impact the estimated SDs [99]. RC linking in PIRLS and TIMSS operates on the 3PL model [98,100,101]. Notably, recalibration methods have also been proposed for determining linking errors in PIRLS/TIMSS [101], as well as in PISA ([69], p. 176ff.), but the two approaches turned out to be different.

RC linking is based on estimated item parameters from the first and the second groups. Item parameters are obtained assuming ${\mu }_1={\mu }_2=0$ and ${\sigma }_1={\sigma }_2=1$ in separate calibrations. The group-specific parameter estimates are defined as:

$$({\widehat{\mathbf{\gamma }}}_0^{\left(1\right)},{\widehat{\mathbf{\gamma }}}_1^{\left(1\right)})=\underset{{\mathbf{\gamma }}_0,{\mathbf{\gamma }}_1}{arg\; max}l(0,1,({\mathbf{\gamma }}_0,{\mathbf{\gamma }}_1);{\mathit{X}}_1)\mathrm{and}$$

(35)

$$({\widehat{\mathbf{\gamma }}}_0^{\left(2\right)},{\widehat{\mathbf{\gamma }}}_2^{\left(2\right)})=\underset{{\mathbf{\gamma }}_0,{\mathbf{\gamma }}_2}{arg\; max}l(0,1,({\mathbf{\gamma }}_0,{\mathbf{\gamma }}_2);{\mathit{X}}_2).$$

(36)

To obtain the mean and the SD of the second group, the data of the first group (i.e., ${\mathit{X}}_1$) are recalibrated using item parameters from the second group (i.e., ${\widehat{\mathbf{\gamma }}}_0^{\left(2\right)}$):

$$(m_1,s_1,{\widehat{\mathit{g}}}_1)=\underset{\mu ,\sigma ,{\mathbf{\gamma }}_1}{arg\; max}l(\mu ,\sigma ,({\widehat{\mathbf{\gamma }}}_0^{\left(2\right)},{\mathbf{\gamma }}_1);{\mathit{X}}_1).$$

(37)

A recalibrated mean $m_1$ and a recalibrated SD $s_2$ are obtained. These two parameters indicate differences between the two groups. Similarly, data of the second group (i.e., ${\mathit{X}}_2$) can be recalibrated using item parameters from the first group (i.e., ${\widehat{\mathbf{\gamma }}}_0^{\left(1\right)}$):

$$(m_2,s_2,{\widehat{\mathit{g}}}_2)=\underset{\mu ,\sigma ,{\mathbf{\gamma }}_2}{arg\; max}l(\mu ,\sigma ,({\widehat{\mathbf{\gamma }}}_0^{\left(1\right)},{\mathbf{\gamma }}_2);{\mathit{X}}_2).$$

(38)

Based on these estimates, the distribution parameters for the second group are defined as:

$${\widehat{\mu }}_2=-s{m}_1,{\widehat{\sigma }}_2=s$$

(39)

where the scaling factor s can take different forms. Note that ${\widehat{\mu }}_2$ relies on the recalibrated mean $m_1$ for the first group and the scaling factor s. The three RC linking methods differ concerning the scaling constant used:

$$\mathrm{Method}\mathrm{RC}1:s=\frac{1}{s_1},\mathrm{Method}\mathrm{RC}2:s=s_2,\mathrm{Method}\mathrm{RC}3:s=\sqrt{\frac{s_2}{s_1}}$$

(40)

The linking methods RC1 and RC2 are asymmetric, while method RC3 relies on both recalibrated SDs. The scaling factor in RC3 linking is defined as the geometric mean of the scaling factors in RC1 and RC2. The definition is motivated by the fact that the impact on recalibrated SDs is symmetrically treated. The linking method RC1 is currently used in PIRLS and TIMSS [96,98] and was used in PISA until 2012 [99]. To our knowledge, linking methods RC2 and RC3 have not yet been investigated in the literature.

3.7. Anchored Item Parameters

The linking method based on anchored item parameters (ANCH; [16,22,102]) estimates the distribution parameters in the second group by fixing the item parameters of the common items to those of the first group. Assume that item parameter estimates in the first group are computed as:

$$({\widehat{\mathbf{\gamma }}}_0,{\widehat{\mathbf{\gamma }}}_1)=\underset{{\mathbf{\gamma }}_0,{\mathbf{\gamma }}_1}{arg\; max}l(0,1,({\mathbf{\gamma }}_0,{\mathbf{\gamma }}_1);{\mathit{X}}_1).$$

(41)

In ANCH linking, ${\mu }_2$ and ${\sigma }_2$ are estimated by maximizing the log-likelihood function while fixing ${\mathbf{\gamma }}_0$ (i.e., ${\mathbf{\gamma }}_0={\widehat{\mathbf{\gamma }}}_0$):

$$({\widehat{\mu }}_2,{\widehat{\sigma }}_2,{\widehat{\mathbf{\gamma }}}_2)=\underset{{\mu }_2,{\sigma }_2,{\mathbf{\gamma }}_2}{arg\; max}l({\mu }_2,{\sigma }_2,({\widehat{\mathbf{\gamma }}}_0,{\mathbf{\gamma }}_2);{\mathit{X}}_2).$$

(42)

It should be noted that RC linking can be considered a variant of ANCH linking because, in RC linking, the distribution parameters of the first group are re-estimated using anchored item parameters from the second group. However, the distribution parameters of the second group are indirectly obtained by transforming the re-estimated distribution parameters of the first group (see Section 3.6). In contrast, ANCH provides an estimate of ${\mu }_2$ and ${\sigma }_2$ directly.

3.8. Concurrent Calibration

Concurrent calibration (CC; [65,70,103]) is based on a multiple-group IRT model and presupposes invariant item groups across groups. The distribution parameters of the second group are determined by maximizing the joint likelihood:

$$({\widehat{\mu }}_2,{\widehat{\sigma }}_2,{\widehat{\mathbf{\gamma }}}_0,{\widehat{\mathbf{\gamma }}}_1,{\widehat{\mathbf{\gamma }}}_2)=\underset{{\mu }_2,{\sigma }_2,{\mathbf{\gamma }}_0,{\mathbf{\gamma }}_1,{\mathbf{\gamma }}_2}{arg\; max}\left\{l(0,1,({\mathbf{\gamma }}_0,{\mathbf{\gamma }}_1);{\mathit{X}}_1)+l({\mu }_2,{\sigma }_2,({\mathbf{\gamma }}_0,{\mathbf{\gamma }}_2);{\mathit{X}}_2)\right\}$$

(43)

In the presence of random DIF, the log-likelihood function in (43) will typically be misspecified. The estimated mean and SD can typically be biased due to this misspecification. It has frequently been pointed out that separate calibration with subsequent linking can be more robust to the presence of DIF than CC [16,54,102]. CC can only be expected to be more efficient than linking based on separate calibrations in small to moderate sample sizes and in the absence of DIF [65]. Notably, CC is more computationally demanding than linking based on separate calibration [65,104].

4. Simulation Study

4.1. Purpose

The purpose of this simulation study was to investigate the performance of linking methods in the two-group case for the 2PL model under different sample sizes, different numbers of items, and different amounts of uniform and nonuniform DIF. Most simulation studies either assume invariant item parameters (i.e., no DIF) or presuppose partial invariance in which only a few item parameters differ between groups (e.g., [63,105,106,107,108,109,110,111,112]). There is a lack of research in the presence of random DIF, although there is some initial work for continuous items [88,89]. Moreover, although recalibration linking is in operational use in LSA studies, they have not yet been systematically compared to alternative linking methods.

We expected from the previous research and our analytical findings in Propositions 1 and 2 that moment-based linking methods could be competitive with the CC and HAE linking methods [54,65]. We did not have specific hypotheses regarding the performance of recalibration linking.

4.2. Design

We simulated a design with two groups and only common items (i.e., no unique items). Data were simulated according to the 2PL model with different amounts of random DIF. In the simulation, it was assumed that there were only common items and no group-specific unique items. The 2PL model with random DIF was simulated according to Equations (3) and (4).

Table A1. Item parameters used in the simulation study.

Item	${\mathit{a}}_{\mathit{i}}$	${\mathit{b}}_{\mathit{i}}$
1	0.95	−0.97
2	0.88	$\phantom{-}$0.59
3	0.75	$\phantom{-}$0.75
4	1.29	−0.79
5	1.28	$\phantom{-}$1.23
6	1.29	−1.10
7	1.25	−0.67
8	0.97	$\phantom{-}$0.20
9	0.73	$\phantom{-}$1.26
10	1.27	$\phantom{-}$0.05
11	1.42	$\phantom{-}$1.22
12	0.75	−0.01
13	0.50	$\phantom{-}$0.20
14	0.81	$\phantom{-}$1.39
15	1.12	$\phantom{-}$0.61
16	0.78	−1.00
17	1.30	−1.58
18	0.70	−1.62
19	1.29	$\phantom{-}$1.06
20	0.74	−0.81

Note.$a_i$ = item discrimination; $b_i$ = item difficulty.

in Appendix F shows item parameters $a_i$ and $b_i$ for 20 items used in the simulation. The first group served as a reference group assuming ${\mu }_1=0$ and ${\sigma }_1=1$, while the distribution parameters for the second group were ${\mu }_2=0.3$ and ${\sigma }_2=1.2$.

In the simulation, four factors were simulated. First, we chose sample sizes $N=500$, 1000, and 5000 (3 factor levels). Second, the item number was either $I=20$ or $I=40$ (2 factor levels). For 40 items, the item parameters from Table A1 were duplicated. The random DIF SD ${\tau }_b$ for item difficulties $b_i$ was 0, 0.1, 0.3, or 0.5 (4 factor levels). The random DIF SD ${\tau }_a$ for item discriminations $a_i$ was 0, 0.15, 0.25 (3 factor levels). In total, there were $3\times 2\times 4\times 3=96$ conditions employed in the simulation.

In total, 1000 datasets were simulated and analyzed in each condition.

4.3. Analysis Methods

The 2PL model was separately estimated in the two groups. Afterward, 13 linking methods (logMM, HAB-log, MM, HAB-nolog, IA2, HAE-asymm, HAE-symm, HAE-joint, RC1, RC2, RC3, ANCH, CC; see Section 3) were applied.

The parameters of interest were the estimated mean ${\widehat{\mu }}_2$ and SD ${\widehat{\sigma }}_2$ for the second group. For the two parameters, the bias and root-mean-squared error (RMSE) were computed. To decrease the dependence of the RMSE on the sample size and the number of items, we computed a relative RMSE for which the RMSE of a linking method was divided by the RMSE of the linking method with the best performance. Hence, this relative RMSE had 100 for the best linking method as its lowest value.

To summarize the contribution of each of the manipulated factors in the simulation, we conducted an analysis of variance (ANOVA) and used a variance decomposition to assess the importance.

Moreover, we classified linking methods as whether or not they showed satisfactory performance in a particular condition. We defined satisfactory performance for the bias if the absolute bias of a parameter (i.e., the estimated mean ${\widehat{\mu }}_2$ or estimated SD ${\widehat{\sigma }}_2$) was smaller than 0.01. In LSA studies such as the Programme for International Student Assessment (PISA), standard errors were about 0.02 or 0.03 for standardized ability variables $\theta$. It should be required that the bias only be a fraction of the variability introduced by the sampling error, which motivated the value of 0.01 as a cutoff. An estimator had satisfactory performance concerning the RMSE if the relative RMSE was smaller than 120. Here, an alternative estimator to the best-performing estimation should not lose too much precision. With an RMSE of 120, the relative-mean-squared error (MSE) was 144 (note that ${1.2}^2=1.44$), which corresponded to a loss in precision of $44\%$. Estimators with worse performance might not be considered as satisfactory in such a situation.

In all the analyses, the statistical software R [113] was used. The R package TAM [114] was used to estimate the 2PL model with marginal maximum likelihood as the estimation method. The linking methods were estimated using dedicated R functions or the existing functionality in the R packages sirt [115] and TAM [114]. The ANOVA model was estimated with the R package lme4 [116].

4.4. Results

In

Table 1. Variance proportions of different factors in the simulation study for the bias and RMSE for the estimated mean ${\widehat{\mu }}_2$ and estimated SD ${\widehat{\sigma }}_2$ for the second group.

Source	${\widehat{\mathit{\mu }}}_2$		${\widehat{\mathit{\sigma }}}_2$
	Bias	RMSE	Bias	RMSE
N	0.3	1.1	0.6	3.9
I	0.0	0.3	0.0	0.0
Meth	10.2$\phantom{1}$	14.9$\phantom{1}$	19.1$\phantom{1}$	0.0
${\tau }_b$	13.0$\phantom{1}$	0.0	0.8	1.8
${\tau }_a$	4.3	9.0	12.3$\phantom{1}$	0.0
N$\times$I	0.0	0.0	0.0	0.0
N$\times$Meth	0.0	3.7	0.8	0.0
N$\times$${\tau }_b$	0.0	2.4	0.0	0.0
N$\times$${\tau }_a$	0.0	0.6	0.0	5.0
I$\times$Meth	0.4	0.1	0.1	0.0
I$\times$${\tau }_b$	0.0	0.1	0.0	0.0
I$\times$${\tau }_a$	0.0	0.0	0.0	0.6
Meth$\times$${\tau }_b$	58.1$\phantom{1}$	13.1$\phantom{1}$	17.5$\phantom{1}$	14.2$\phantom{1}$
Meth$\times$${\tau }_a$	8.2	12.1$\phantom{1}$	47.7$\phantom{1}$	13.2$\phantom{1}$
${\tau }_a$$\times$${\tau }_b$	0.0	4.1	0.0	17.7$\phantom{1}$
N$\times$I$\times$Meth	0.0	0.0	0.1	0.0
N$\times$I$\times$${\tau }_b$	0.0	0.2	0.0	0.0
N$\times$I$\times$${\tau }_a$	0.0	0.1	0.0	0.4
N$\times$Meth$\times$${\tau }_b$	0.2	7.5	0.0	4.2
N$\times$Meth$\times$${\tau }_a$	0.0	4.0	0.0	9.1
N$\times$${\tau }_a$$\times$${\tau }_b$	0.1	10.0$\phantom{1}$	0.0	13.8$\phantom{1}$
I$\times$Meth$\times$${\tau }_b$	0.5	0.0	0.0	0.4
I$\times$Meth$\times$${\tau }_a$	0.1	0.0	0.2	1.1
I$\times$${\tau }_a$$\times$${\tau }_b$	0.1	0.3	0.0	0.7
Meth$\times$${\tau }_a$$\times$${\tau }_b$	1.0	10.1$\phantom{1}$	0.1	8.2
Residual	3.7	6.4	0.6	5.7

Note.N = sample size; I = number of items; Meth = linking method; ${\tau }_a$ = standard deviation of DIF effects in item discriminations $a_i$; ${\tau }_b$ = standard deviation of DIF effects in item difficulties $b_i$. Percentage values larger than 1.0 are printed in bold.

, the variance decomposition of the ANOVA is presented. All terms up to three-way interactions are included. From the size of the residual variance, it can be concluded that the first three orders are sufficient to capture the most important factors in the simulation.

It turned out that sample size (N) and the number of items (I) were only of minor importance in the first three-order terms in ANOVA. However, the linking method, as well as the size of random DIF were important factors. Importantly, the random DIF SD (${\tau }_b$) in item difficulties $b_i$ had a large influence on the estimated means, while random DIF SD (${\tau }_a$) in item discriminations $a_i$ strongly impacted the estimated SDs.

Due to these observations, we decided to provide aggregated results for three important groups of cells in the simulation. First, we aggregated the results for six conditions with no DIF (NODIF; ${\tau }_b=0$ and ${\tau }_a=0$). Because there were two estimated parameters (mean and SD), this resulted in aggregation across twelve measures for the bias and RMSE, respectively. Second, we considered all 18 conditions with uniform DIF (UDIF; ${\tau }_b>0$ and ${\tau }_a=0$). Third, we provide summaries across all 48 conditions with nonuniform DIF (NUDIF; ${\tau }_a>0$).

In

Table 2. Summary of the satisfactory performance of linking methods for the absolute bias and RMSE across parameters (mean ${\widehat{\mu }}_2$ and standard deviation ${\widehat{\sigma }}_2$) and conditions.

	Bias			RMSE
	NODIF	UDIF	NUDIF	NODIF	UDIF	NUDIF
logMM	100	$\phantom{1}$97	94	100	100	45
HAB	100	$\phantom{1}$97	94	100	100	44
MM	100	$\phantom{1}$94	95	$\phantom{1}$92	100	72
HAB-nolog	100	$\phantom{1}$94	96	100	100	78
IA2	$\phantom{1}$75	$\phantom{1}$78	$\phantom{1}$8	100	100	$\phantom{1}$4
HAE-asymm	100	$\phantom{1}$42	42	100	$\phantom{1}$61	78
HAE-symm	100	$\phantom{1}$97	94	100	$\phantom{1}$61	81
HAE-joint	100	$\phantom{1}$42	60	100	$\phantom{1}$42	61
RC1	$\phantom{1}$83	$\phantom{1}$78	16	100	$\phantom{1}$61	29
RC2	$\phantom{1}$83	$\phantom{1}$78	$\phantom{1}$8	100	$\phantom{1}$61	48
RC3	100	$\phantom{1}$94	96	100	$\phantom{1}$61	79
ANCH	$\phantom{1}$83	$\phantom{1}$78	13	100	$\phantom{1}$61	48
CC	100	$\phantom{1}$50	45	100	$\phantom{1}$33	46

Note. DIF = differential item functioning; NODIF = no DIF; UDIF = uniform DIF; NUDIF = nonuniform DIF; logMM = log-mean-mean linking; HAB = Haberman linking with logarithmized item discriminations; MM = mean-mean linking; HAB-nolog = Haberman linking with untransformed item discriminations; IA2 = invariance alignment with power $p=2$; HAE-asymm = asymmetric Haebara linking; HAE-symm = symmetric Haebara linking; HAE-joint = Haebara linking with joint item parameters; RC = recalibration linking(see Equation (40)); ANCH = anchored item parameters; CC = concurrent calibration. Values smaller than 70 are printed in bold.

, the performance and the linking methods are summarized across these three groups of conditions by classifying all linking methods as either satisfactory or nonsatisfactory. Table 2 shows the proportion of conditions in which a linking method provided satisfactory results. In the absence of DIF (columns “NODIF”), all methods performed well in most of the conditions. Notably, IA2, the asymmetric recalibration methods RC1 and RC2, as well as ANCH provided biases in some instances. In the case of uniform DIF (columns “UDIF”), HAE-asymm, HAE-joint and CC cannot be recommended in terms of the bias. Moreover, only moment-based methods (logMM, HAB, MM, HAB-nolog) can be recommended in terms of the RMSE. Finally, in the presence of nonuniform DIF (columns “NUDIF”), moment-based methods, as well as HAE-symm and RC3 were satisfactory in terms of the bias. If the RMSE is considered as an additional criterion, utilizing linking methods MM, HAB-nolog, HAE-symm, and RC3 can be suggested.

In

Table 3. Bias and RMSE for mean ${\widehat{\mu }}_2$ and standard deviation ${\widehat{\sigma }}_2$ for the second group for a sample size $N=1000$ and $I=40$ items as a function of the type of differential item functioning and linking method.

	Bias			RMSE
	NODIF	UDIF	NUDIF	NODIF	UDIF	NUDIF
	${\mathbf{\tau }}_{\mathit{b}}=\mathbf{0}$	${\mathbf{\tau }}_{\mathit{b}}=\mathbf{0.5}$	${\mathbf{\tau }}_{\mathit{b}}=\mathbf{0.5}$	${\mathbf{\tau }}_{\mathit{b}}=\mathbf{0}$	${\mathbf{\tau }}_{\mathit{b}}=\mathbf{0.5}$	${\mathbf{\tau }}_{\mathit{b}}=\mathbf{0.5}$
	${\mathbf{\tau }}_{\mathit{a}}=\mathbf{0}$	${\mathbf{\tau }}_{\mathit{a}}=\mathbf{0}$	${\mathbf{\tau }}_{\mathit{a}}=\mathbf{0.25}$	${\mathbf{\tau }}_{\mathit{a}}=\mathbf{0}$	${\mathbf{\tau }}_{\mathit{a}}=\mathbf{0}$	${\mathbf{\tau }}_{\mathit{a}}=\mathbf{0.25}$
	Mean${\widehat{\mu }}_2$
logMM	$\phantom{-}$0.000	$\phantom{-}$0.007	$\phantom{-}$0.008	108.2	104.4	106.1
HAB	$\phantom{-}$0.000	$\phantom{-}$0.007	$\phantom{-}$0.008	108.2	104.4	106.1
MM	$\phantom{-}$0.000	$\phantom{-}$0.007	$\phantom{-}$0.007	108.1	103.7	104.7
HAB-nolog	$\phantom{-}$0.001	$\phantom{-}$0.007	$\phantom{-}$0.007	108.5	103.5	104.5
IA2	−0.001	$\phantom{-}$0.001	$\phantom{-}$0.045	103.2	107.5	133.3
HAE-asymm	−0.002	−0.030	−0.032	102.3	100.0	100.0
HAE-symm	−0.001	$\phantom{-}$0.002	$\phantom{-}$0.005	102.7	105.0	105.2
HAE-joint	−0.002	$\phantom{-}$0.067	$\phantom{-}$0.064	100.9	136.1	132.4
RC1	−0.001	$\phantom{-}$0.001	$\phantom{-}$0.028	100.2	104.8	120.5
RC2	−0.006	−0.004	−0.022	100.0	104.0	100.1
RC3	−0.003	−0.001	$\phantom{-}$0.002	100.1	103.9	109.4
ANCH	−0.003	−0.004	−0.021	101.4	104.2	103.9
CC	−0.002	$\phantom{-}$0.095	$\phantom{-}$0.109	101.3	149.2	157.7
	Standard Deviation${\widehat{\sigma }}_2$
logMM	$\phantom{-}$0.000	$\phantom{-}$0.003	$\phantom{-}$0.008	110.2	112.6	128.9
HAB	$\phantom{-}$0.000	$\phantom{-}$0.003	$\phantom{-}$0.008	110.2	112.6	129.4
MM	−0.001	$\phantom{-}$0.001	$\phantom{-}$0.005	108.5	109.4	107.7
HAB-nolog	$\phantom{-}$0.001	$\phantom{-}$0.002	$\phantom{-}$0.007	100.0	100.0	100.0
IA2	$\phantom{-}$0.009	$\phantom{-}$0.009	$\phantom{-}$0.147	113.2	111.6	197.9
HAE-asymm	−0.002	−0.120	−0.134	107.2	378.8	185.6
HAE-symm	$\phantom{-}$0.001	−0.003	$\phantom{-}$0.003	108.3	233.7	119.9
HAE-joint	−0.001	$\phantom{-}$0.020	$\phantom{-}$0.029	107.5	317.0	146.6
RC1	$\phantom{-}$0.006	$\phantom{-}$0.008	$\phantom{-}$0.105	109.8	243.8	174.5
RC2	−0.009	−0.008	−0.097	108.5	217.2	148.3
RC3	−0.002	$\phantom{-}$0.000	$\phantom{-}$0.002	106.6	228.3	110.2
ANCH	−0.009	−0.008	−0.097	108.5	217.2	148.3
CC	−0.001	$\phantom{-}$0.015	$\phantom{-}$0.029	107.4	220.4	129.0

Note: DIF = differential item functioning; NODIF = no DIF; UDIF = uniform DIF; NUDIF = nonuniform DIF; ${\tau }_a$ = standard deviation of DIF effects in item discriminations $a_i$; ${\tau }_b$ = standard deviation of DIF effects in item difficulties $b_i$; logMM = log-mean-mean linking; HAB = Haberman linking with logarithmized item discriminations; MM = mean-mean linking; HAB-nolog = Haberman linking with untransformed item discriminations; IA2 = invariance alignment with power $p=2$; HAE-asymm = asymmetric Haebara linking; HAE-symm = symmetric Haebara linking; HAE-joint = Haebara linking with joint item parameters; RC = recalibration linking(see Equation (40)); ANCH = anchored item parameters; CC = concurrent calibration. Absolute biases larger than 0.02 are printed in bold. RMSE values larger than 120 are printed in bold.

, the bias and RMSE for the mean ${\widehat{\mu }}_2$ and the SD ${\widehat{\sigma }}_2$ of the second group for $N=1000$ students and $I=40$ items are shown. It can be seen that all linking methods performed well in the absence of DIF (see columns “NODIF”). In the case of uniform DIF (columns “UDIF”), HAE-asymm, HAE-joint, and CC produced nonsatisfactory results in terms of the bias for the mean or the SD. Interestingly, the RMSE for the SD was much larger for HAE, RC, ANCH, and CC than the moment-based methods logMM, HAB, MM, and logMM. In the case of nonuniform DIF (column “NUDIF”), only moment-based methods (except IA2) and HAE-symm and the newly proposed recalibration linking method RC3 produced satisfactory results. Furthermore, note that the RMSE for the SD was larger for logMM and HAB compared to MM and HAB-nolog. This finding can be explained by the fact that the data-generating model for DIF was an additive model that operated on nontransformed item discriminations and favored MM and HAB-nolog. To conclude, linking methods that are not moment-based can only be recommended as a default linking method when the mean is of interest and there is an absence of random DIF. It depends on the extent of UDIF (i.e., size of ${\tau }_b$) whether the additional variance introduced by moment-based methods is compensated by smaller biases.

5. Empirical Example: Linking PISA 2006 and PISA 2009 for Austria

5.1. Method

In order to illustrate the consequences of different scaling models (i.e., 1PL and 2PL models), as well as different linking methods (see Section 3), we analyzed the data for Austrian students from the PISA study conducted in 2006 (PISA 2006; [95]) and 2009 (PISA 2009; [117]). In

Table 4. Sample information and descriptive results for PISA 2006 and PISA 2009 Austria.

Domain	N		I		$\mathbf{M}$		$\mathbf{SD}$
	P06	P09	P06	P09	P06	P09	P06	P09
Mathematics	3784	4575	$\phantom{1}$48	35	506.8	495.9	$\phantom{1}$96.8	$\phantom{1}$96.1
Reading	2646	6585	$\phantom{1}$27	99	491.2	470.3	107.7	100.1
Science	4927	4577	103	53	511.7	494.3	$\phantom{1}$97.3	101.8

Note:$\mathrm{M}$ = mean; $\mathrm{SD}$ = standard deviation; P06 = PISA 2006; P09 = PISA 2009; N = number of students; I = number of items; $\mathrm{M}$ = mean; $\mathrm{SD}$ = standard deviation.

, the sample sizes (i.e., N) and item numbers (i.e., I) used are presented. There were common items in the two PISA studies (Mathematics: $I_0=35$; Reading: $I_0=27$; Science: $I_0=53$). Moreover, the officially reported means and SDs for Austrian students in PISA 2006 and PISA 2009 are displayed. Note that PISA 2006 and PISA 2009 employed the 1PL model and utilized the MM linking method with a subsequent recalibration linking (method RC1; [117]).

In both PISA studies, we included only those students who received a test booklet with at least one item in a respective domain. For simplicity, all polytomous items (i.e., items with maximum scores larger than one) were dichotomously recoded, with only the highest category being recoded as correct. The 1PL and the 2PL model were used as scaling models, and student weights were taken into account. In total, 13 linking methods (logMM, HAB-log, MM, HAB-nolog, IA2, HAE-asymm, HAE-symm, HAE-joint, RC1, RC2, RC3, ANCH, CC; see Section 3) were applied. In the linking procedure, the distribution parameters of the first study (PISA 2006) were fixed (i.e., ${\mu }_1=0$, ${\sigma }_1=0$), and the distribution parameters of the second study (PISA 2009) were estimated. The two ability distributions for the three domains (and the distribution parameters) were linearly transformed such that the mean equaled the officially reported mean and the SD equaled the officially reported SD in PISA 2006 in the respective domain. For example, for Mathematics, it holds that ${\widehat{\mu }}_1=\mathrm{M}=506.8$ and ${\widehat{\sigma }}_1=\mathrm{SD}=96.8$ in PISA 2006 for all linking methods for the 1PL and the 2PL model.

5.2. Results

In

Table 5. Trend estimate for Austrian students in average achievement from PISA 2006 to PISA 2009.

Method	Mathematics		Reading		Science
	1PL	2PL	1PL	2PL	1PL	2PL
logMM	−15.5	−12.4	−5.8	−6.3	−14.7	−16.8
HAB	−15.5	−12.4	−5.8	−6.3	−14.7	−16.8
MM	−15.5	−12.4	−5.8	−6.3	−14.7	−16.7
HAB-nolog	−15.5	−12.3	−6.0	−6.3	−14.5	−16.6
IA2	−15.5	−15.9	−5.8	−6.1	−14.7	−11.6
HAE-asymm	−14.4	−14.6	−4.9	−6.4	−14.2	−15.9
HAE-symm	−14.6	−15.0	−5.0	−6.6	−14.2	−15.7
HAE-joint	−13.5	−14.1	−4.1	−5.0	−13.9	−14.0
RC1	−14.3	−14.5	−4.4	−5.1	−14.0	−13.2
RC2	−14.3	−14.3	−4.3	−5.0	−14.2	−12.9
RC3	−14.3	−14.4	−4.4	−5.0	−14.1	−13.1
ANCH	−14.4	−15.7	−4.5	−5.4	−14.5	−14.1
CC	−14.3	−14.9	−4.3	−5.3	−14.2	−13.6
M	−14.8	−14.1	−5.0	−5.8	−14.3	−14.7
SD	$\phantom{-}$$\phantom{1}$0.7	$\phantom{-}$$\phantom{1}$1.3	$\phantom{-}$0.7	$\phantom{-}$0.6	$\phantom{-}$$\phantom{1}$0.3	$\phantom{-}$$\phantom{1}$1.8
Min	−15.5	−15.9	−6.0	−6.6	−14.7	−16.8
Max	−13.5	−12.3	−4.1	−5.0	−13.9	−11.6

Note. logMM = log-mean-mean linking; HAB = Haberman linking with logarithmized item discriminations; MM = mean-mean linking; HAB-nolog = Haberman linking with untransformed item discriminations; IA2 = invariance alignment with power $p=2$; HAE-asymm = asymmetric Haebara linking; HAE-symm = symmetric Haebara linking; HAE-joint = Haebara linking with joint item parameters; RC = recalibration linking(see Equation (40)); ANCH = anchored item parameters; CC = concurrent calibration.

, the trend estimates for Austria from PISA 2006 and PISA 2009 are shown. For the linearly transformed scores, the trend estimate is given as $\widehat{\Delta }={\widehat{\mu }}_2-{\widehat{\mu }}_1$. There was a significant and large negative trend for Mathematics and Science, while the trend in Reading turned out to be smaller. Across both models (1PL and 2PL) and linking methods, the average trend in Mathematics was $\mathrm{M}=-14.4$ ($\mathrm{SD}=1.1$, $\mathrm{Range}=3.6$). There were slight differences between the 1PL and the 2PL models for the moment-based linking methods (i.e., logMM, HAB, MM, HAB-nolog). Notably, the variation of estimates across linking methods was larger for the 2PL model ($\mathrm{SD}=1.3$) than for the 1PL model ($\mathrm{SD}=0.7$). The trend in Reading was smaller ($\mathrm{M}=-5.4$) and less variable ($\mathrm{SD}=0.8$, $\mathrm{Range}=2.4$) than in Mathematics. Finally, the trend in Science was also strongly negative ($\mathrm{M}=-14.5$). The variability ($\mathrm{SD}=1.3$, $\mathrm{Range}=5.2$) was mainly caused by the variability in linking methods of the 2PL model. In contrast, linking methods for the 1PL model were very similar ($\mathrm{SD}=0.3$, $\mathrm{Range}=0.8$).

In

Table 6. Standard deviation for Austrian students in PISA 2009. for domains Mathematics, Reading and Science for the 1PL and the 2PL model as a function of the linking method.

Method	Mathematics		Reading		Science
	1PL	2PL	1PL	2PL	1PL	2PL
logMM	97.7	98.3	$\phantom{1}$98.6	103.2	103.2	106.8
HAB	97.7	98.3	$\phantom{1}$98.6	103.2	103.2	106.8
MM	97.7	98.7	$\phantom{1}$98.6	103.8	103.2	106.9
HAB-nolog	97.9	99.3	$\phantom{1}$94.6	102.0	103.9	108.1
IA2	97.7	99.5	$\phantom{1}$98.6	104.6	103.2	109.2
HAE-asymm	94.1	95.0	102.6	105.4	105.0	107.5
HAE-symm	95.0	96.2	103.1	105.9	105.3	107.8
HAE-joint	95.0	95.7	105.1	107.5	104.7	107.4
RC1	96.0	96.9	103.1	107.2	103.9	108.6
RC2	96.0	95.6	$\phantom{1}$99.9	106.2	104.7	105.9
RC3	96.0	96.3	101.5	106.7	104.3	107.2
ANCH	96.0	95.6	$\phantom{1}$99.9	106.2	104.7	105.9
CC	95.9	96.7	101.3	106.4	104.1	107.5
M	96.3	97.1	100.4	105.2	104.1	107.4
SD	$\phantom{1}$1.2	$\phantom{1}$1.5	$\phantom{1}$$\phantom{1}$2.7	$\phantom{1}$$\phantom{1}$1.7	$\phantom{1}$$\phantom{1}$0.7	$\phantom{1}$$\phantom{1}$0.9
Min	94.1	95.0	$\phantom{1}$94.6	102.0	103.2	105.9
Max	97.9	99.5	105.1	107.5	105.3	109.2

Note. logMM = log-mean-mean linking; HAB = Haberman linking with logarithmized item discriminations; MM = mean-mean linking; HAB-nolog = Haberman linking with untransformed item discriminations; IA2 = invariance alignment with power $p=2$; HAE-asymm = asymmetric Haebara linking; HAE-symm = symmetric Haebara linking; HAE-joint = Haebara linking with joint item parameters; RC = recalibration linking(see Equation (40)); ANCH = anchored item parameters; CC = concurrent calibration.

, the estimated SD ${\widehat{\sigma }}_2$ in PISA 2009 is displayed for the 1PL and the 2PL models as a function of the linking method. Interestingly, the variability across linking methods turned out to be larger than for the mean estimate. Relatively large differences of the 1PL model and the 2PL model were observed for Reading (1PL: $\mathrm{M}=100.4$, 2PL: $\mathrm{M}=105.2$) and Science (1PL: $\mathrm{M}=104.1$, 2PL: $\mathrm{M}=107.4$). Note that the differences between the 1PL and the 2PL model were particularly pronounced for the IA2 method. The difference between the two scaling models might be explained by different average item discriminations for common items and unique items. Reading was a minor domain in PISA 2006 (no unique items) and a major domain in PISA 2009 in which a large set of unique items were introduced. Science was a major domain in PISA 2006 (many unique items) and a minor domain in PISA 2009 (no unique items). In contrast, Mathematics was a minor domain in PISA 2006 and PISA 2009, and there was only a small number of unique items in PISA 2006 and no unique items in PISA 2009. These findings indicate that an adjustment for the average differences in item discriminations has to be conducted to avoid biased estimates of the means and SDs when a misspecified 1PL model is employed (see [99]).

6. Discussion

In this article, several linking methods for two groups were evaluated for the 2PL model in the case of normally distributed random DIF effects with zero means for item difficulties and item discriminations. Somehow surprisingly and contrary to the recommendations in the literature, moment-based linking methods (mean-mean and log-mean-mean, as well as Haberman linking) performed best in terms of the bias and RMSE. The unbiasedness of moment-based methods in the case of many items was expected due to our consistency results for the estimators presented in Section 3. When the primary criterion is the bias, symmetric Haebara (HAE-symm) linking and the newly proposed recalibration method RC3 can only be recommended among the nonmoment-based methods. In contrast, the commonly used asymmetric Haebara (HAE-asymm) linking and the recalibration linking RC1 used in PIRLS and TIMSS had substantially worse performance. Furthermore, note that concurrent calibration—which incorrectly assumes invariant item parameters across groups—and the anchored item parameters method provided biased estimates and cannot be recommended in operational use. Concurrent calibration can only achieve the promised highest efficiency [67,70] in small-to-moderate-sized samples, the absence of random DIF, and a correctly specified IRT model. If data in educational large-scale assessment studies were indeed to follow a random DIF distribution, the currently used linking methods (concurrent calibration, recalibration linking RC1) could be replaced by better alternatives as proposed in this study (moment-based linking, recalibration linking RC3). Of course, the extent of the bias and loss in the precision of the current methods depends on the variance of the DIF effects and can vary from study to study.

Our study assumed that the utilized scaling model (i.e., the 2PL model) was correctly specified. This assumption might be unrealistic in practice, and data could have been generated with much more complex item response functions [118,119,120,121,122] or multidimensional IRT models [123,124]. The performance of the linking methods for misspecified IRT models [125,126] in the presence of random DIF might be an exciting topic for future research [127,128].

In this article, we only considered random DIF effects with a normal distribution and zero means. For relevance in practical applications, it might be interesting for future research to study the DIF effects under different data-generating models. First, random DIF could be simulation in a partial invariance situation in which only a few items have DIF effects, while the majority of DIF effects do not have DIF effects. Second, the case of normally distributed random DIF and partial invariance could occur in tandem. DIF effects could be simulated from a mixture distribution in which the first class includes normally distributed DIF effects with zero means, while the second class of smaller proportion includes outlying and large DIF effects. Due to the presence of outliers, the average of DIF effects will typically differ from zero. For this kind of DIF effects, robust linking methods must be employed for removing these outlying items from group comparisons [65,71,89,94,107].

In this article, we only studied linking in the two-group case. In the case of multiple groups, different linking methods can be employed [65,84,93,129,130]. However, the findings from two groups are likely to generalize to the multiple group case if the linking were to be performed based on sequences of pairwise linking approaches [131,132,133]. Comparing the performance of pairwise linking approaches and simultaneous linking of multiple groups would be an exciting topic for future research.

Our findings are likely to have even more impact on vertical scaling in which groups constitute time points or grades in the school career [134]. In this situation, differences in the means and SDs between groups are expected to be larger, and the consequences of choosing an inappropriate linking method can be much more pronounced [128,135,136,137,138]. As pointed out by an anonymous reviewer, vertical linking poses more assumptions than cross-sectional linking because younger test-takers could incorporate guessing strategies if they had a chance to respond to harder items designed for older test-takers (see also [25]).

It should be noted that we did not investigate the computation of standard errors for the linking methods. There is a rich literature that derives standard error formulas for linking due to sampling of persons (e.g., [85,104,132,139,140,141]). In addition, variability in estimated group means and SDs due to selecting items has been studied as linking errors in the literature [66,72,99,142,143,144,145]. It might be interesting in future research to investigate standard errors that reflect these sources of uncertainty [72,84,132,146]. Procedures that rely on resampling of persons and items will likely correctly reflect uncertainty due to persons and items in the parameters of interest.