Why Randomized HR Evaluations May Mislead Managers: The Role of Treatment–Trait Interactions in Outcome Construction

Abstract

This paper examines whether randomized evaluations can fail to identify causal effects when outcomes include interactions between treatment and unobserved characteristics. We show that even under random assignment, standard regression estimators do not necessarily recover the structural causal effect if outcomes contain non-separable interaction terms between treatment and latent characteristics. When outcomes contain such non-separable interaction terms, the estimated treatment effect reflects interaction components embedded in the outcome construction and may fail to recover the structural policy parameter. We derive conditions under which unbiased identification is restored, highlighting the critical role of additive separability. The results provide a theoretical foundation for understanding when randomized evaluations may yield misleading conclusions in managerial and policy contexts.

1. Introduction

Managers increasingly rely on experimental evidence to guide decisions about which human resource (HR) programs to adopt, expand, or discontinue. In many organizations, randomized evaluations are used to compare mentoring schemes, training programs, or job assignment policies, and the resulting estimates often inform resource allocation and performance benchmarking. Such evaluations typically involve randomly assigning employees to different HR interventions (e.g., training, mentoring, or job assignments) and comparing their subsequent performance outcomes. Throughout the paper, we assume that randomized evaluations rely on the assumption that treatment assignment is statistically independent of workers’ unobserved characteristics, ensuring that treatment assignment is exogenous in the standard sense. Despite their growing importance, these evaluations do not always provide clear guidance. Randomized HR evaluations can sometimes be misleading for managers, especially when performance measures reflect interactions between treatment and worker characteristics. In particular, these interaction terms may arise from non-separable functional relationships in the outcome equation, complicating the interpretation of estimated effects. In such cases, estimated treatment effects may not isolate the intrinsic effectiveness of the program, as they also reflect interaction-driven variation that depends on workforce composition, potentially biasing managerial decisions. Our analysis is also related to broader concerns regarding the external validity and context dependence of experimental evidence, as treatment effects may vary with underlying workforce characteristics.

While treatment assignment may be fully randomized and independent of worker characteristics, the estimated treatment effect may nevertheless fail to recover the structural policy parameter when outcome construction embeds interaction terms between treatment and latent characteristics.

The central point of this paper is not that randomization fails, but that randomization alone is insufficient for identifying the structural policy effect when outcomes include non-separable interaction components, a feature that is common in practical organizational settings.

In practice, results from randomized HR evaluations are often used not only to assess whether a program works but also to compare alternative interventions, rank organizational units, and guide resource allocation. For example, a firm might expand mentoring programs that appear highly effective or discontinue training schemes with weak estimated impacts. Implicit in these decisions is the assumption that randomized evaluation delivers unbiased estimates of program effectiveness.

This paper examines a precise identification problem: whether standard regression estimators in randomized evaluations recover the structural policy parameter (i.e., the direct effect of the intervention) when outcome construction embeds interactions between treatment and latent worker characteristics.

Our analysis is also related to the literature on the interpretation of treatment effects and structural policy parameters (Heckman & E. Vytlacil, 2005; Heckman & E. J. Vytlacil, 2007), as well as to recent work on external validity and policy extrapolation (Deaton, 2010; Vivalt, 2020). While these studies emphasize the distinction between reduced-form treatment effects and economically meaningful parameters, we highlight a distinct mechanism: even under random assignment, regression coefficients may fail to recover the structural policy effect when outcome construction embeds interaction terms between treatment and latent characteristics.

Even when treatment is randomly assigned, the regression coefficient on treatment need not isolate the structural policy effect, because outcome measures may mechanically incorporate treatment–type interactions. Our objective is not to challenge the validity of randomized evaluations, but to clarify the conditions under which the estimated coefficient coincides with the underlying structural parameter.

Building on this identification perspective, we develop a theoretical framework to analyze a limitation that arises even within a single experimental setting. In particular, when performance metrics embed interactions between treatment and worker characteristics, conventional evaluation methods may not isolate the direct effect of an intervention. In reality, employee performance depends not only on policy exposure but also on how well the intervention matches an employee’s latent type—for instance, whether a mentoring style suits their learning preferences or whether a training program complements their existing skills.

When outcome measures incorporate such interaction-based match components, estimated treatment effects may reflect not only the intrinsic effectiveness of the program but also workforce composition. Consequently, differences in estimated impacts across firms, units, or time periods may stem more from variation in employee characteristics than from program quality. This raises the risk that managers could overestimate the superiority of a program simply because it aligns with a higher proportion of employees who benefit most.

To clarify this, we develop a simple analytical framework in which performance depends on both a direct policy effect and an interaction-based match component interacting with worker heterogeneity. Our analysis shows that while random assignment ensures independence between treatment and worker type, it does not guarantee independence between treatment and constructed outcome components. As a result, standard regression estimates may be biased when match quality is non-separable in treatment and worker characteristics. Importantly, unbiased recovery of the direct policy effect requires a stronger condition: additive separability between treatment and worker type in the outcome-generating process.

This paper contributes to the literature in economics and management science, while also providing insights for HR practitioners. To illustrate the practical magnitude of this problem, we present a simulation based on a targeted mentoring setting, in which employee performance serves as the outcome measure and incorporates both direct policy effects and interaction-based match components. The results indicate that estimated treatment effects can vary systematically with workforce composition, even when the underlying policy effect remains constant. From a managerial perspective, this suggests that program rankings based on estimated impacts can be unstable or misleading if differences in employee composition are not carefully considered.

This mechanism implies that managers may systematically favor programs that perform well under the current workforce composition rather than those that are intrinsically more effective, leading to a distortion in resource allocation.

The contribution of this paper is threefold. First, we show that even under perfect random assignment, the regression coefficient on treatment does not necessarily identify the structural policy parameter when outcome construction embeds non-separable interactions between treatment and latent characteristics. Second, we establish a transparent analytical condition under which the structural policy parameter is identified, thereby clarifying when conventional evaluation methods remain valid. Third, we demonstrate how evaluation bias translates into distorted managerial decisions, particularly in program comparison and resource allocation. Our objective is not to critique the use of randomized evaluations, but to clarify the conditions under which their results can be correctly interpreted in organizational settings.

2. Related Literature

2.1. Personnel Economics and HR Interventions

A large amount of the literature in personnel economics examines how compensation schemes, organizational practices, and assignment policies affect worker performance (Lazear, 1995; Lazear & Oyer, 2012). Empirical studies have documented that HR practices such as performance pay, team-based incentives, and structured management systems can substantially influence productivity (Huselid, 1995; Ichniowski et al., 1997).

More recently, randomized evaluations have been used to assess specific workplace interventions. Bloom et al. (2015) provide experimental evidence on the productivity effects of working from home. Hoffman et al. (2018) analyze how evaluation and managerial practices interact with worker characteristics in shaping performance outcomes. These studies demonstrate the growing role of experimental methods in personnel research.

However, although these contributions emphasize causal identification through random assignment, they typically abstract from the possibility that outcome measures themselves embed assignment-based match components, i.e., interaction-based components between treatment and latent worker characteristics.

While the literature establishes the importance of HR practices for productivity, it provides limited direction on how such evidence should be used in managerial decision-making, particularly when performance measures embed interaction effects.

Recent work in HR analytics and data-driven personnel management further highlights the increasing reliance of firms on quantitative evaluation tools to guide managerial decisions (Boudreau & Cascio, 2017; Levenson, 2018; Minbaeva, 2018). Marler and Boudreau (2017) provide a comprehensive review of HR analytics, emphasizing its role in linking workforce data to strategic decision-making. Angrave et al. (2016) critically examine the limitations of HR analytics practices, noting that data-driven evaluations may yield misleading conclusions when underlying assumptions are not carefully considered. Similarly, Rasmussen and Ulrich (2015) discuss how HR analytics transforms managerial decision processes by integrating data analysis into organizational strategy.

These contributions underscore the growing importance of evaluation frameworks in HR management but also point to potential pitfalls when performance metrics are not properly aligned with underlying causal mechanisms. In particular, they highlight that managerial reliance on quantitative metrics can lead to misinterpretation when evaluation systems embed complex interactions between interventions and workforce characteristics.

In particular, this paper contributes to the HR analytics literature by clarifying a previously underexplored identification issue: even when firms rely on randomized evaluation, the structure of performance metrics themselves may distort managerial inference.

The present paper complements the literature by providing a formal analytical framework that clarifies how such interaction structures affect identification and interpretation in HR evaluation. However, while the literature emphasizes the importance of data-driven decision-making, it has paid limited attention to how the structure of performance metrics themselves may affect causal interpretation, which is the focus of the present paper.

Importantly, the present paper contributes to the literature by highlighting a fundamental identification issue that arises when performance metrics—widely used in HR analytics and managerial decision-making—embed interaction structures between interventions and employee characteristics.

This perspective situates our contribution within the growing literature on evidence-based HR management, where the interpretation of evaluation results plays a central role in organizational decision-making.

2.2. Matching and Assignment Models

The importance of worker–firm and worker–task matching has long been recognized in labor economics. Jovanovic (1979) models job matching as a process of learning about match quality, while Sattinger (1993) surveys assignment models that explain wage and productivity dispersion. More recent work highlights how complementarities between worker ability and task characteristics shape equilibrium outcomes (Eeckhout & Kircher, 2011; Shimer & Smith, 2000).

These models emphasize that productivity depends on interactions between worker type and assignment. The present paper draws on this insight but focuses on a distinct question: how such complementarities affect empirical identification in randomized HR evaluations when interaction-based match components are embedded in outcome construction.

From a managerial perspective, these complementarities are often reflected in performance evaluation systems and HR analytics frameworks, where measured outcomes depend on the alignment between interventions and worker characteristics.

2.3. Interpretation of Treatment Effects and Structural Parameters

This paper is closely related to the literature on the interpretation of treatment effects and structural policy parameters. Heckman and E. Vytlacil (2005); Heckman and E. J. Vytlacil (2007) emphasize that reduced-form treatment effect estimates may not coincide with economically meaningful structural parameters. Related contributions further develop the structural interpretation of treatment effects (Carneiro et al., 2011). In a related spirit, we show that even under random assignment, regression coefficients do not necessarily recover the structural policy effect when outcome construction embeds interaction terms between treatment and latent characteristics.

Our analysis is also connected to the literature on external validity and policy extrapolation (Deaton, 2010; Vivalt, 2020), which highlights that estimated effects may depend on the context in which they are measured. Related concerns have also been raised regarding site selection and contextual dependence in program evaluation (Allcott, 2015). While the literature focuses on cross-context variation in estimated effects, we instead emphasize a within-context identification problem arising from the structure of outcome variables.

2.4. Heterogeneous Treatment Effects and Identification

The econometric literature has long recognized that treatment effects may vary across individuals. Imbens and Angrist (1994) formalize local average treatment effects under heterogeneous responses, while Heckman et al. (1997) analyze selection bias and heterogeneous impacts in program evaluation. More recently, Athey and Imbens (2017) and Chernozhukov et al. (2018) develop methods for estimating heterogeneous treatment effects using modern econometric tools. Recent advances in machine learning methods further extend the literature by enabling flexible estimation of heterogeneous effects (Athey & Wager, 2019; Knaus et al., 2021). Recent developments in difference-in-differences designs also highlight challenges in identifying causal effects under complex treatment structures (Callaway & Sant’Anna, 2021), particularly when treatment effects are heterogeneous or evolve over time.

The present paper differs from the literature in focus. Rather than proposing new estimators for heterogeneous effects, we show that when outcome construction embeds interactions between treatment and latent types, conventional regressions may yield biased estimates of the direct policy effect even under random assignment. Accordingly, the paper highlights a specification issue that arises from the structure of the outcome variable itself.

However, the literature does not address how such interaction structures affect the interpretation of evaluation results in practical HR settings. In such settings, evaluation results are used to inform managerial decisions.

Distinction from Heterogeneous Treatment Effects

At first glance, the mechanism analyzed here may appear closely related to the heterogeneous treatment effects (HTE) literature. However, the present argument is conceptually distinct. The HTE literature studies variation in causal effects across individuals and focuses on identifying or estimating average or conditional treatment effects under heterogeneity (Athey & Imbens, 2017; Imbens & Angrist, 1994). In contrast, this paper highlights a specification issue arising from the structure of the outcome variable itself. Even when the average treatment effect is well-defined and consistently estimated under random assignment, the regression coefficient on treatment need not recover the direct structural policy parameter when outcome construction embeds interaction-based match components. The bias identified here therefore does not stem from selection or non-random assignment, but from mechanical covariance between treatment and interaction terms embedded in performance measurement.

In this sense, the paper addresses a decomposition problem rather than an estimation problem under heterogeneity. Importantly, the bias identified here arises even when the average treatment effect is well-defined and consistently estimable, and therefore cannot be resolved by standard heterogeneity-robust methods, including approaches designed to estimate heterogeneous treatment effects.

Formally, the average treatment effect can be written as

$$E[Y_i\left(1\right)-Y_i\left(0\right)]={\gamma }_0+\delta E[m(1,{\theta }_i)-m(0,{\theta }_i)],$$

where $Y_i\left(1\right)$ and $Y_i\left(0\right)$ denote the potential performance outcomes of worker i with and without the HR intervention, respectively. Here, ${\gamma }_0$ represents the direct policy effect, $m(T_i,{\theta }_i)$ captures interaction-based match quality as a function of treatment $T_i$ and worker type ${\theta }_i$, and $\delta$ scales the contribution of match quality.

This expression implies that the regression coefficient on $T_i$ does not isolate ${\gamma }_0$ when the interaction term is embedded in the outcome measure.

Taken together, the existing literature provides important insights into HR practices, matching, and treatment heterogeneity, but offers limited guidance on how firms should interpret evaluation results when performance measures embed interactions between policy exposure and worker characteristics. In practice, organizations increasingly rely on HR analytics and composite performance metrics to evaluate interventions, compare programs, and guide managerial decisions. In such settings, the distinction between intrinsic policy effectiveness and composition-driven effects becomes critical. This paper contributes to bridging this gap by linking econometric identification to practical evaluation design and managerial decision-making.

3. HR Assignment Framework

3.1. Targeted HR Interventions and Worker Heterogeneity

This section develops a simple framework to illustrate how evaluation bias can arise in practical HR settings.

Many HR policies are explicitly targeted. Firms frequently design mentoring programs, training modules, or job assignments intended to benefit workers with particular latent characteristics, such as career orientation, skill profile, or motivational type. Let $T_i\in \{0,1\}$ denote assignment to such an HR intervention for worker i. We assume that treatment is randomly assigned, as in a standard randomized evaluation design.

Workers differ in unobserved characteristics ${\theta }_i$, representing latent type. These characteristics may capture dimensions such as learning aptitude, task preference, or compatibility with particular forms of supervision. We assume

$$T_i\perp {\theta }_i,$$

implying that treatment assignment is statistically independent of worker type, consistent with random assignment.

In many HR environments, performance outcomes depend not only on the direct effect of policy exposure but also on the quality of the match between the intervention and the worker’s latent type. For example, the effectiveness of a mentoring program may depend on whether the mentor’s style aligns with the employee’s learning orientation, or whether a training module complements the worker’s skill base.

In practice, such heterogeneity is often unobserved by the analysts but implicitly reflected in performance evaluation systems used by firms.

3.2. Outcome Construction with Interaction-Based Match

To formalize this structure, suppose that worker performance is given by

$$Y_i=\alpha +{\gamma }_0{T}_i+\delta m(T_i,{\theta }_i)+{\epsilon }_i,$$

(1)

where

• ${\gamma }_0$ denotes the direct policy effect;
• $m(T_i,{\theta }_i)$ represents interaction-based match quality;
• $\delta$ scales the contribution of match quality;
• $E[{\epsilon }_i\mid T_i,{\theta }_i]=0$.

The key feature of Equation (1) is that match quality may depend jointly on policy exposure and worker type. In targeted HR interventions, this is natural: the intervention is often designed precisely to interact with particular worker characteristics. This formulation is consistent with common HR evaluation settings, where measured performance (e.g., composite evaluation scores) combines direct productivity effects with assessment components that depend on worker–intervention fit.

Suppose that the researcher estimates the conventional evaluation regression derived from Equation (1), omitting the interaction-based match component:

$$Y_i=\alpha +\gamma T_i+u_i,$$

(2)

3.3. Evaluation Bias Under Targeted Assignment

The probability limit of the ordinary least squares estimator in Equation (2) is

$$plim\widehat{\gamma }={\gamma }_0+\delta {\displaystyle \frac{Cov(T_i,m(T_i,{\theta }_i))}{\mathrm{Var}\left(T_i\right)}}.$$

(3)

Random assignment ensures statistical independence between $T_i$ and ${\theta }_i$. However, as shown in Equation (3), it does not imply independence between $T_i$ and the composite term $m(T_i,{\theta }_i)$, because match quality may be partially defined by treatment itself.

Evaluation bias arises whenever

$$Cov(T_i,m(T_i,{\theta }_i))\ne 0.$$

This covariance can be positive in targeted environments where the intervention is particularly effective for a subset of worker types, leading to systematic overestimation of program effectiveness. In such cases, the estimated treatment coefficient reflects not only the intrinsic effectiveness of the policy (${\gamma }_0$), but also the prevalence of worker types for whom the intervention generates strong match gains.

From a managerial perspective, this implies that estimated program effectiveness may be systematically overstated or understated depending on workforce composition.

3.4. Identification Condition

A transparent condition ensures consistent estimation of the direct policy effect. If match quality satisfies the additive separability condition in Equation (4),

$$m(T_i,{\theta }_i)=a\left(T_i\right)+b\left({\theta }_i\right),$$

(4)

then

$$Cov(T_i,m(T_i,{\theta }_i))=0,$$

and the conventional regression consistently estimates ${\gamma }_0$. Equation (4) therefore provides a transparent sufficient condition for unbiased identification of the direct policy effect.

Additive separability implies that the incremental contribution of match quality does not depend on the interaction between policy exposure and worker type. In contrast, non-separability corresponds to complementarity between intervention and latent characteristics—a feature that is common in targeted HR policies.

Thus, independence between treatment and worker heterogeneity is not, by itself, sufficient to guarantee unbiased policy evaluation in practical HR settings. What matters is whether the outcome construction embeds interactions between assignment and type.

This condition is unlikely to hold in many practical HR settings where interventions are explicitly designed to target specific worker types.

4. Simulation of Managerial Evaluation Outcomes

This simulation represents a stylized HR evaluation environment in which employee performance is used as the primary outcome measure and treatment effects may depend on latent worker characteristics. This structure captures typical HR evaluation systems in which performance metrics combine objective output measures and subjective assessments, both of which may depend on the match between interventions and worker characteristics.

To illustrate how randomized HR evaluations can mislead managerial decision-making, we simulate a mentoring program targeted at high-potential employees. The objective is not only to quantify evaluation bias, but also to examine how standard evaluation practices may lead managers to draw incorrect conclusions about program effectiveness when workforce composition varies.

4.1. Data-Generating Process

Match quality is specified as

$$m(T_i,{\theta }_i)=T_i\mathbf{1}\{{\theta }_i=1\},$$

capturing the idea that mentoring improves performance primarily for high-potential employees. The performance equation follows

$$Y_i=\alpha +{\gamma }_0{T}_i+\delta m(T_i,{\theta }_i)+{\epsilon }_i,$$

with ${\gamma }_0=0.5$, $\delta =1.0$, and ${\epsilon }_i\sim N(0,1)$.

Let ${\pi }_1=Pr({\theta }_i=1)$ denote the share of high-potential employees in the workforce. The remaining three types each have probability ${\pi }_2={\pi }_3={\pi }_4=(1-{\pi }_1)/3$. We vary ${\pi }_1$ from 0.25 to 1.0 to examine how workforce composition affects evaluation bias.

The theoretical bias equals

$$\mathrm{Bias}=\delta {\pi }_1,$$

implying that the estimated treatment effect increases proportionally with the prevalence of high-potential employees.

The simulation uses $N=2000$ observations and $R=3000$ replications.

4.2. Results

Table 1. Monte Carlo performance under targeted mentoring.

Share of High- Potential Workers (${\mathit{\pi }}_1$)	Theoretical Bias ($\mathit{\delta }{\mathit{\pi }}_1$)	Simulated Bias (Naive)	RMSE (Naive)	RMSE (Correct)
0.25	0.250	0.250	0.254	0.049
0.32	0.320	0.320	0.323	0.049
0.39	0.390	0.391	0.393	0.051
0.46	0.460	0.460	0.462	0.053
0.53	0.530	0.529	0.531	0.056
0.60	0.600	0.599	0.601	0.060
0.67	0.670	0.670	0.671	0.063
0.74	0.740	0.741	0.743	0.069
0.81	0.810	0.810	0.811	0.079
0.88	0.880	0.880	0.881	0.099
0.95	0.950	0.951	0.953	0.144

Notes: The table reports Monte Carlo simulation results based on the data-generating process described in Section 4. The sample size is $N=2000$ with $R=3000$ replications. The naive estimator omits the interaction-based match component, while the correct estimator includes it. Theoretical bias is given by $\delta {\pi }_1$.

illustrates how managerial assessments of program effectiveness vary with workforce composition. It reports the theoretical bias ($\delta {\pi }_1$), the simulated bias of the naive estimator that omits match quality, and the root mean squared error (RMSE) of both the naive and correctly specified estimators. The results confirm the analytical characterization: for each level of workforce composition, the simulated bias of the naive estimator closely aligns with the theoretical prediction $\delta {\pi }_1$. The close alignment between analytical and simulated bias indicates that the distortion arises directly from the interaction structure rather than from small-sample artifacts.

As the share of high-potential workers (${\pi }_1$) increases, the magnitude of bias rises proportionally, even though the intrinsic policy effect ${\gamma }_0$ remains constant across scenarios. As a result, identical mentoring programs would appear systematically more effective in organizations with a higher share of high-potential employees. Consequently, managers relying on standard evaluation estimates may incorrectly rank programs or organizational units, leading to misleading performance comparisons.

The RMSE of the naive estimator also increases with ${\pi }_1$, reflecting the growing covariance between treatment and interaction-based match quality. In contrast, the correctly specified estimator remains centered around zero bias and exhibits substantially smaller RMSE across all composition levels. This further implies that evaluation uncertainty increases with workforce composition, making decision-making more difficult precisely in settings where the intervention appears most effective.

Figure 1 shows that the estimated effect systematically overstates program effectiveness as the share of targeted workers increases, leading to a misleading ranking of program performance across organizations. The simulated bias of the naive estimator closely follows the analytical prediction $\delta {\pi }_1$, increasing linearly with the share of high-potential workers. The correctly specified estimator remains centered around zero across all scenarios. These results confirm that the bias arises directly from the interaction structure embedded in outcome construction.

Figure 2 displays the corresponding RMSE results. As the share of high-potential workers increases, the dispersion of the naive estimator rises substantially, whereas the correctly specified estimator remains comparatively stable.

The results are robust to alternative treatment assignment probabilities and to different variance levels of the disturbance term (not reported for brevity).

More broadly, concerns about outcome measurement and interpretation have been emphasized in recent work (Baker et al., 2022; Bond & Lang, 2019).

4.3. Illustrative Example: Composite Performance Evaluation

To illustrate the practical relevance of the mechanism, consider a firm that evaluates employees using a composite performance score. Let $S_i$ denote the annual evaluation score for worker i, constructed as a weighted index of output quantity, project completion, and mentor assessment:

$$S_i=w_1{Q}_i+w_2{P}_i+w_3{M}_i,$$

where $Q_i$ denotes output quantity, $P_i$ denotes project completion or task performance, $M_i$ denotes mentor or supervisor assessment, and $w_1,w_2,w_3$ are the corresponding weights.

Composite evaluation systems of this kind are common in practice, where firms frequently aggregate multiple performance dimensions into a single score used for promotion or compensation decisions.

Consider a setting in which participation in a mentoring program directly improves productivity by ${\gamma }_0$, while also affecting the mentor assessment component $M_i$ through an interaction with worker type. For instance, mentors may provide higher ratings to workers whose learning style aligns with the mentoring format.

Under this structure, the evaluation score can be written as

$$S_i=\alpha +{\gamma }_0{T}_i+\delta T_i·g\left({\theta }_i\right)+{\epsilon }_i,$$

where $T_i$ denotes participation in the mentoring program, ${\gamma }_0$ denotes the direct productivity effect, $g\left({\theta }_i\right)$ denotes type-dependent responsiveness in evaluation (e.g., mentor assessment), and $\delta$ scales the contribution of this interaction.

Even under random assignment of the mentoring program, the composite score embeds an interaction between treatment and latent type. A regression of $S_i$ on $T_i$ alone therefore combines the direct productivity effect with the match-dependent evaluation component. The resulting estimate may reflect the aggregation of heterogeneous responses induced by the evaluation formula rather than the intrinsic effectiveness of the policy.

As a result, managers may attribute higher evaluation scores to superior program design, when they actually reflect the interaction between the evaluation system and workforce composition.

5. Practical Implications for HR Evaluation

The analysis yields several implications for managerial decision-making in HR evaluation. Although the framework developed in this paper is theoretical, it directly maps into managerial decision-making in practical HR evaluation settings.

Beyond interpretation, the bias identified in this paper has direct implications for organizational decision-making. When estimated treatment effects reflect workforce composition rather than intrinsic program effectiveness, managers may systematically favor interventions that align with the current workforce rather than those that are inherently more productive.

This may lead to inefficient resource allocation, as programs targeted toward more prevalent worker types receive disproportionate investment. In dynamic settings, such misallocation may also distort organizational learning, as firms update beliefs based on biased performance metrics. From an economic perspective, this implies that evaluation bias can translate into systematic misallocation of organizational resources.

In light of these considerations, managers should carefully assess whether performance measures incorporate interaction-based match components. In many practical settings, evaluation metrics such as composite performance scores implicitly embed interactions between interventions and worker characteristics. When such interactions are present, estimated treatment effects may not reflect intrinsic program effectiveness.

Managers should avoid relying solely on average treatment effects when comparing HR interventions. Differences in estimated effects across units or firms may be driven by workforce composition rather than differences in program quality.

Evaluation designs should explicitly account for treatment–type interaction. In practice, this may involve incorporating observable proxies for worker heterogeneity, conducting subgroup analysis, or testing the robustness of results across different workforce compositions.

Managers should exercise caution when using evaluation results for resource allocation decisions. Programs targeted at more prevalent worker types may appear more effective and thus receive disproportionate investment, even if their intrinsic impact is not superior.

Finally, firms should recognize that randomized assignment, while essential for causal identification, does not by itself guarantee reliable managerial inference when outcome construction embeds interaction between assignment and worker heterogeneity. Careful evaluation design and interpretation are therefore critical in HR analytics.

6. Conclusions

This paper addresses a subtle yet consequential identification issue in evaluating targeted HR policies under random assignment. In practice, performance outcomes depend not only on direct policy exposure but also on interaction-based match components that vary with latent worker characteristics. When such interactions are embedded in outcome construction, conventional evaluation regressions may fail to isolate the direct policy effect. Importantly, this paper does not challenge the use of randomized HR evaluations but rather clarifies how their results should be interpreted when outcome measures incorporate such interaction effects.

Our analysis confirms that random assignment ensures independence between treatment and underlying worker heterogeneity. However, it does not guarantee independence between treatment and composite outcomes that embed assignment-dependent components. We identify a transparent condition—additive separability between policy exposure and worker type within the match function—that allows consistent estimation of the intrinsic policy effect. When this condition does not hold, estimated treatment effects can partially reflect workforce composition rather than the true effectiveness of the intervention.

Monte Carlo simulations illustrate that the magnitude of evaluation bias can grow systematically with the prevalence of targeted worker types, even when the intrinsic policy effect remains unchanged. These simulations are designed to mimic a stylized HR evaluation setting in which employee performance is used as the primary outcome measure and treatment effectiveness depends on latent worker characteristics. This underscores the importance of accounting for match structures when comparing HR interventions across firms, units, or workforce compositions. Misinterpretation in these contexts can have real managerial consequences, such as misallocation of resources or suboptimal program selection.

The framework presented here is deliberately parsimonious, abstracting from dynamic adjustments, equilibrium responses, and learning effects. It provides a theoretical foundation that complements practical evaluation settings commonly encountered in HR management. Its aim is not to question the value of randomized evaluations but to clarify conditions under which standard specifications may fail in targeted assignment settings. By highlighting the role of treatment–type interactions in outcome construction, this work seeks to improve both the empirical design and interpretation of randomized HR policy evaluations.

From an economic perspective, the identification problem highlighted in this paper implies that evaluation bias can lead to systematic misallocation of resources when managerial decisions are based on composition-driven performance metrics. In environments where firms rely heavily on data-driven evaluation, such distortions may accumulate over time, affecting both productivity and organizational efficiency. This may also distort organizational learning and long-run resource allocation.

From a managerial perspective, the key takeaway is that evaluation results should be interpreted in the context of workforce composition and performance measurement design. In this sense, the paper contributes not only to econometric identification but also to the HR analytics literature by clarifying how performance measurement design affects managerial interpretation of evaluation results. Managers comparing HR programs across units or over time should recognize that observed differences in estimated treatment effects may reflect composition-driven interactions rather than genuine differences in program efficacy. Careful interpretation, coupled with attention to outcome construction, is essential to avoid systematic misallocation of organizational resources.