Synthetic Control and Inference

Abstract

We examine properties of permutation tests in the context of synthetic control. Permutation tests are frequently used methods of inference for synthetic control when the number of potential control units is small. We analyze the permutation tests from a repeated sampling perspective and show that the size of permutation tests may be distorted. Several alternative methods are discussed.

1. Introduction

Synthetic control method, proposed and discussed by Abadie and Gardeazabal (2003) and Abadie et al. (2010), is a very useful way of conducting comparative studies when exact matches are unavailable. Estimation of treatment effects usually takes the form of comparing outcomes between the treated unit and the control unit. Common sense suggests that, for the comparison to be meaningful, the control unit needs to be similar to the treated unit in the absence of the treatment in various dimensions. Such a requirement may not be satisfied in many observational studies. In some cases, availability of panel data makes such comparisons reasonable, the difference-in-differences method being a very well-known example. The difference-in-differences method requires a very specific set of assumptions, i.e., the common trend assumption, which may not be plausible for many applications. The synthetic control method offers a sensible generalization of the difference-in-differences. The synthetic control is a linear combination of the potential control outcomes, where the weights are manufactured by analyzing the pre-intervention outcomes.

For the purpose of statistical inference with synthetic control, i.e., confidence interval and hypothesis testing, various versions of placebo tests are often adopted. The idea underlying the placebo tests is the usual permutation tests, where the critical value of a test statistic is computed under all possible permutations of the “treatment” assignments in the control units.

The idea of permutation test is very intuitive and attractive. Applying the synthetic control method to every potential control unit presumably allows researchers to assess the distribution of a test statistic under the null hypothesis of no treatment effects, and the inference is seemingly exact in the sense that the burden of asymptotic approximation can be obviated.

The purpose of this paper is very specific. We ask whether the permutation test is a reasonable idea in the context of the synthetic control method, and argue that the intuitive appeal of the permutation test is misplaced. The validity of permutation tests usually requires certain symmetry assumption, which is often violated in the context of synthetic control studies. Using Monte Carlo simulations, we document the size distortion of the permutation tests. We also discuss a few alternative methods of inference.

Alberto Abadie kindly pointed out that the placebo test in synthetic control is often based on randomization inference idea, under which the symmetry restriction is built-in, while our analysis is predicated from the usual random sampling perspective, which leads to the violation of symmetry. This perspective is shared with an anonymous referee, who notes that (i) the synthetic control literature uses permutation tests in the context of design-based inference, and, as such, the permutation tests have exact size; (ii) the present article shows that permutation tests may not have correct size under a different mode of inference based on repeated sampling, although interpreting the permutation tests in the previous literature as tests based on repeated sampling would be incorrect; and (iii) the present article also proposes some alternatives that are valid in a repeated sampling setting. It would be useful to understand the exact mechanism through which the difference between the two perspectives manifests itself. The same referee points out that the present paper adopts a setting where $T_0\to \infty$, while the original litereature assumes fixed $T_0$.

2. Placebo Test and Synthetic Control

In this section, we provide a brief discussion of the placebo test in the context of the synthetic control method. We begin with an overview of the synthetic control, borrowing heavily from discussions in Abadie et al. (2010) and Doudchenko and Imbens (2016). We then move on to describe the placebo test, and point out the importance of the symmetry assumption. We argue that the symmetry assumption is violated in general for placebo tests using linear combinations of outcomes, such as synthetic control. We conclude this section that such violation should be expected in general even when a normalized version of the test statistic is adopted.

We start with the overview of the synthetic control method. Consider a panel data with $J+1$ cross sectional units observed over the time periods $t=1,\dots ,T$. Units $j=1,\dots ,J$ are the control units that receive the treatment in none of the time periods. The unit $j=0$ receives no treatment in periods $1,\dots ,T_0$, and receives active treatment in time periods $t=T_0+1,\dots ,T$. For simplicity, we will often assume that $T=T_0+1$. The outcome variable $Y_{j,t}$ is such that $Y_{j,t}=Y_{j,t}\left(1\right)$ if the jth unit receives treatment in time t, and $Y_{j,t}=Y_{j,t}\left(0\right)$ otherwise. Obviously,

$$\begin{array}{cc}\hfill Y_{j,t}& =Y_{j,t}\left(0\right),j=1,\dots ,J;t=1,\dots ,T,\hfill \\ \hfill Y_{0,t}& =Y_{0,t}\left(0\right),t=1,\dots ,T_0,\hfill \\ \hfill Y_{0,t}& =Y_{0,t}\left(1\right),t=T_0+1,\dots ,T.\hfill \end{array}$$

The idea underlying the synthetic control is that if there were some weights1 ${\widehat{\omega }}_1,\dots ,{\widehat{\omega }}_J$ such that

$$Y_{0,t}\approx \sum _{j=1}^J{\widehat{\omega }}_j{Y}_{j,t}$$

(1)

during the pre-intervention periods ($t=1,\dots ,T_0$). Then, ${\sum }_{j=1}^J{\widehat{\omega }}_j{Y}_{j,t}$ can be used as a (synthetic) control for $Y_{0,t}$ during the post-intervention periods ($t=T_0+1,\dots ,T$). Abadie et al. (2010) and Doudchenko and Imbens (2016) discuss various methods of finding the $\widehat{\omega }$’s so that the requirement in Equation (1) is satisfied. We analyze the weights and the nature of approximation from the asymptotic perspective where $T_0\to \infty$. Note that a special case of the estimator discussed by Abadie et al. (2010, p. 496) solves

$$\underset{{\omega }_1,\dots ,{\omega }_J}{min}{\left({\overline{Y}}_0-\sum _{j=1}^J{\omega }_j{\overline{Y}}_j\right)}^2.$$

(2)

Under our interpretation, ${\overline{Y}}_j=T_0^{-1}{\sum }_{t=1}^{T_0}{Y}_{j,t}$ above is an estimator of $E\left[Y_{j,t}\right]$.

Consider the linear factor structure2 as in Abadie et al. (2010):

$$Y_{j,t}=Y_{j,t}\left(0\right)={\alpha }_j+{\theta }_t+{\gamma }_j^{\prime }{\delta }_t+{ϵ}_{j,t},t=1,\dots ,T_0.$$

(3)

Suppose that ${\theta }_t$, ${\delta }_t$, and ${ϵ}_{j,t}$ satisfy strict stationarity. Without loss of generality, we also assume that $E\left[{\delta }_t\right]=0$ and $E\left[{ϵ}_{j,t}\right]=0$. We would have ${\overline{Y}}_j\to {\alpha }_j+E\left[{\theta }_t\right]$ in probability as $T_0\to \infty$. Assuming that ${\omega }_{\ast j}$ satisfies

$${\alpha }_0+E\left[{\theta }_t\right]=\sum _{j=1}^J{\omega }_{\ast j}\left({\alpha }_j+E\left[{\theta }_t\right]\right),$$

(4)

we can understand that the population version of the synthetic control ${\sum }_{j=1}^J{\omega }_{\ast j}{Y}_{j,t}$ is such that the difference $Y_{0,T_0+1}\left(0\right)-{\sum }_{j=1}^J{\omega }_{\ast j}{Y}_{j,T_0+1}\left(0\right)$ is designed to have a mean zero.

Our $T_0\to \infty$ asymptotic interpretation is not the only possible one. Doudchenko and Imbens (2016) provide an in-depth analysis of many possible methods. Our interpretation, however, is helpful for two reasons. First, it makes a concrete interpretation of $\widehat{\omega }$s as estimates of some pseudo-parameter, say ${\omega }_{\ast }$’s, along with analytic expressions of the ${\omega }_{\ast }$’s, which makes it easy to understand the potential pitfalls of permutation methods afterwards. Second, it helps us to motivate alternative methods of inference exploiting time series variation.

We now discuss how placebo tests can be used in the context of synthetic control. For this purpose, we first present a summary of the placebo tests/permutation tests. The tests are motivated to deal with the case where the number of the treated is small and the number of controls is relatively large. In order to focus on the salient feature of the tests, we will consider an extreme case and assume that there is only one treated unit.

The basic intuition underlying the general placebo test can be gleaned by examining a standard textbook case of randomized treatments. Suppose that there is cross sectional data with $J+1$ units, where the units $j=1,\dots ,J$ are the control units and the unit $j=0$ receives the active treatment. A reasonable estimator of the treatment effect is the difference $Y_0-\overline{Y}$, where $Y_0$ is the outcome of the unit $j=0$, and $\overline{Y}=J^{-1}{\sum }_{j=1}^J{Y}_j$ denotes the average of the outcomes of the controls. Suppose that we are interested in testing whether the treatment had impact. Given that there is only one treated unit, the standard t-test comparing the difference of the mean outcomes is not applicable. On the other hand, common sense suggests that we may implement such a test by “assigning” each control unit to fictitious treatment. More precisely, one can estimate the empirical distribution of $Y_k-{\left(J-1\right)}^{-1}{\sum }_{j\ne k}{Y}_j$ for $k=1,\dots ,J$, and use it as if it were the distribution of the treatment effect under the null hypothesis.3

Implementation of the placebo test with synthetic control requires a bit more notation. First let $\widehat{\omega }={\left({\widehat{\omega }}_1,\dots ,{\widehat{\omega }}_J\right)}^{\prime }$ denote the estimator of ${\omega }_{\ast }={\left({\omega }_{\ast 1},\dots ,{\omega }_{\ast J}\right)}^{\prime }$. Although we will use the method of exact balancing later in our Monte Carlo simulations, we do not need to restrict ourselves to this particular estimator. For now, we can view $\widehat{\omega }$ as an output from a blackbox and let ${\omega }_{\ast }$ denote its probability limit as $T_0\to \infty$. Second, let ${\widehat{\omega }}^{\left(-k\right)}$ denote the outcome of the same blackbox except that we use the kth unit as the outcome of the treated unit, and $Y_{j,t}$ with $j\ne k$ as our control units. The placebo test then uses the empirical distribution of $Y_{k,T_0+1}-{\sum }_{j\ne k}{\widehat{\omega }}_j^{\left(-k\right)}{Y}_{j,T_0+1}$ for $k=1,\dots ,J$ as if it were the distribution of the treatment effect under the null hypothesis of no treatment effect. If the estimated effect $Y_{0,T_0+1}-{\sum }_{j=1}^J{\widehat{\omega }}_j{Y}_{j,T_0+1}$ belongs to the extreme tails of the empirical distribution, it is understood to be the evidence that the null hypothesis is incorrect.

In order to understand the size property of the placebo test, it helps to recall that the placebo test is a version of the permutation test, which requires for its validity what may be called the symmetry assumption. For review of this property, we will borrow the short discussion in Canay et al. (2017).4 Suppose that a researcher observes a vector of observations X, whose joint distribution is P. The objective is to test whether $P\in {\mathbf{P}}_0$, where ${\mathbf{P}}_0$ is a collection of probability distributions such that the distribution of X is equal to that of $gX$ for every g in $\mathbf{G}$, where $\mathbf{G}$ is a finite collection of transformations. The permutation test has the exact size if, for the test statistic $T\left(X\right)$, the critical value is taken from the distribution of $T\left(gX\right)$ for every g in $\mathbf{G}$. In the context of the placebo test above, one can understand X to be the vector $\left(Y_1,\dots ,Y_J\right)$, and $gX$ to be the permutation of the Ys.

We note that the symmetry is not mathematically obvious in the context of synthetic control. In order for the permutation test to be valid, it is necessary for the distribution of $Y_{0,T_0+1}-{\sum }_{j=1}^J{\omega }_{\ast j}{Y}_{j,T_0+1}$ and those of $Y_{k,T_0+1}-{\sum }_{j\ne k}{\omega }_{\ast j}^{\left(-k\right)}{Y}_{j,T_0+1}$ for $k=1,\dots ,J$ to be identical. Even for the relatively simple model in Equation (3), the nature of the synthetic control is such that the symmetry does not naturally follow. Using the restriction in Equation (4), we may write

$$Y_{0,T}-\sum _{j=1}^J{\omega }_{\ast j}{Y}_{j,T}=\left(\right.{\theta }_T-E\left[{\theta }_T\right]\left.\right)\left(1-\sum _{j=1}^J{\omega }_{\ast j}\right)+{\left({\gamma }_0-\sum _{j=1}^J{\omega }_{\ast j}{\gamma }_j\right)}^{\prime }{\delta }_T+\left({ϵ}_{0,T}-\sum _{j=1}^J{\omega }_{\ast j}{ϵ}_{j,T}\right).$$

(5)

Even if the first two terms on the right-hand side of Equation (5) were identically equal to zero over the permutations, we believe that the third term is not likely to satisfy the symmetry property. This is because we believe that under the further restriction that the $ϵ$’s have a finite variance, the term can be symmetric only when they are normally distributed.

We show that normality is necessary if the distribution of the error term ${ϵ}_{0,T}-{\omega }_{\ast }^{\prime }{ϵ}_T$ in Equation (5), where ${ϵ}_T={\left({ϵ}_{1,T},\dots ,{ϵ}_{J,T}\right)}^{\prime }$, is to be symmetric up to normalization.5 Suppose that ${ϵ}_{0,T},\dots ,{ϵ}_{J,T}$ are i.i.d., and their common distribution is such that the variance is finite and the characteristic function does not disappear. If $\omega$ is a nontrivial function of $\alpha$s and $\gamma$s, then symmetry over the permutations requires that the marginal distributions of ${ϵ}_{k,T}-{\sum }_{j\ne k}{\omega }_j^{\left(-k\right)}{ϵ}_{j,T}$ for $k=0,\dots ,J$ should remain invariant over all possible ${\omega }^{\left(-k\right)}$s. Without loss of generality, we can focus on the distribution of ${ϵ}_{0,T}-{\omega }^{\prime }{ϵ}_T$, and conclude that the symmetry requires that there exists a random variable Y such that the distribution of ${ϵ}_{0,T}-{\omega }^{\prime }{ϵ}_T$ is the same as that of $cY$ for some scalar c. Because the standard deviation of ${ϵ}_{0,T}-{\omega }^{\prime }{ϵ}_T$ is proportional to $\sqrt{1+{\omega }^{\prime }\omega }$, we may without loss of generality take $c=\sqrt{1+{\omega }^{\prime }\omega }$. This implies that the distribution of ${\omega }^{\prime }{ϵ}_T$ only depends on ${\omega }^{\prime }\omega$. In other words, for $\omega \ne \tilde{\omega }$ such that ${\omega }^{\prime }\omega ={\tilde{\omega }}^{\prime }\tilde{\omega }$, the distribution of ${\omega }^{\prime }{ϵ}_T$ is identical to that of ${\tilde{\omega }}^{\prime }{ϵ}_T$. In particular, let all components of $\tilde{\omega }$ be zero except for the first one. Then, the distribution of ${\omega }^{\prime }{ϵ}_T$ is identical to that of ${\tilde{\omega }}^{\prime }{ϵ}_T={\tilde{\omega }}_1{ϵ}_{1,T}=\sqrt{{\tilde{\omega }}^{\prime }\tilde{\omega }}{ϵ}_{1,T}$. This implies that ${ϵ}_{j,T}$ should have a stable distribution.6 Because the only stable distribution with a finite variance is the normal distribution, we should conclude that normality is a necessary condition of the symmetry (up to normalization). Note that the third term in Equation (5) arises in an ideal situation where the weights $\omega$ do not need to be estimated and the first two terms completely disappear. Our analysis suggests that even if we normalize the third term by its standard deviation, the symmetry requires normal distribution. The necessity of normality assumption is about any linear combination so it applies a fortiori to synthetic control.

3. Monte Carlo

The discussion at the end of the previous section casts doubt on the placebo test, even for the simple case where the first two terms in Equation (5) can be ignored. In order to understand the roles that the first two terms may play, we adopt Monte Carlo simulations. We try to find data generating processes (DGPs hereafter) that generate a large amount of size distortions. This is helpful in understanding the potential problem of the placebo test from the uniformity perspective; after all, the mathematical definition of the “size” of a test is the maximum probability of rejection under the null, and here the null hypothesis is a composite hypothesis where the only requirement on the DGP is that the treatment effect is zero, which allows many possibilities on the terms in Equation (5). For this purpose, we found it most convenient to work with the first two terms in Equation (5), although we acknowledge that there may be other important sources of size distortion that we have not explored. Since the last paragraph of Section 2 showed that normalization does not abate the symmetry requirement, we examine the importance of the first two terms in Equation (5) using a more natural statistic. The version of the synthetic control that we use in the Monte Carlo is the method of exact balancing, the population version of which minimizes ${\sum }_{j=1}^J{\omega }_j^2$ subject to $E\left[Y_{0,t}\left(0\right)\right]={\sum }_{j=1}^J{\omega }_{j}E\left[Y_{j,t}\left(0\right)\right]$ and $1={\sum }_{j=1}^J{\omega }_j$.7

The method of exact balancing may not be an ideal version of the synthetic control, but it reflects a certain ambiguity in the method of synthetic control. In the factor model in Equation (3), it is impossible to find weights $\omega$ such that $Y_{0,t}={\sum }_{j=1}^J{\omega }_j{Y}_{j,t}$ for every $t=1,\dots ,T_0$, if $T_0$ is large enough, as long as ${ϵ}_{j,t}$ is continuously distributed. In other words, the condition (2) in Abadie et al. (2010) is incompatible with the factor model unless $Var\left({ϵ}_{j,t}\right)=0$. The assumption $Var\left({ϵ}_{j,t}\right)=0$ has at least two implications.8 First, the weights ${\omega }_j$ can be estimated without error with sufficiently large $T_0$. Second, the distribution of the permutation test would have the point mass at zero, and as such, there is no reason to conduct any test. Both implications are questionable. In any case, under the assumption $Var\left({ϵ}_{j,t}\right)=0$, the weights can be estimated (without error) by the method of least squares that minimizes ${\sum }_{t=1}^{T_0}{\left(Y_{0,t}-{\sum }_{j=1}^J{\omega }_j{Y}_{j,t}\right)}^2$. If the assumption $Var\left({ϵ}_{j,t}\right)=0$ is violated, the method of least squares would be subject to a version of measurement error problem; the true regressor there is ${\alpha }_j+{\theta }_t+{\gamma }_j^{\prime }{\delta }_t$ in Equation (3), and the $Y_{j,t}$ plays the role of a regressor with measurement error ${ϵ}_{j,t}$.9 Note that such a problem is avoided by the method of exact balancing.

We consider the method of exact balancing in this section not because it is necessarily an ideal version of the synthetic control, but because it is a convenient way of examining the impact of the first two terms in Equation (5). As mentioned at the beginning of this section, our analysis at the end of the previous section suggests that the placebo test may have a problem even when these two terms are dismissed, and the purpose of our Monte Carlo exercise is to focus on the potential impact of these two terms.

For our Monte Carlo analysis, we adopted a simplified version of the factor model in Equation (3) such that (i) ${\theta }_t\sim \mathcal{N}\left(0,1\right)$; (ii) ${\delta }_t$ is a scalar; (iii) ${\delta }_t\sim \mathcal{N}\left(0,{\sigma }_{\delta }^2\right)$; (iv) ${ϵ}_{j,t}\sim \mathcal{N}\left(0,{\sigma }_{ϵ}^2\right)$ is i.i.d. over j and t. In matrix notation, our estimator $\widehat{\omega }$ solves

$$min{\omega }^{\prime }\omega \mathrm{s}.\mathrm{t}.(\mathrm{i}){\overline{Y}}^{\prime }\omega ={\overline{Y}}_0,(\mathrm{ii}){\ell }^{\prime }\omega =1,$$

(6)

where ℓ is a vector of ones. Because $E\left[Y_{j,t}\right]={\alpha }_j$, we can see that the population counterpart ${\omega }_{\ast }$ solves

$$min{\omega }^{\prime }\omega \mathrm{s}.\mathrm{t}.\left(\mathrm{i}\right){\alpha }^{\prime }\omega ={\alpha }_0,\left(\mathrm{ii}\right){\ell }^{\prime }\omega =1,$$

where $\alpha ={\left({\alpha }_1,\dots ,{\alpha }_J\right)}^{\prime }$.

We now write

$$Y_{0,T}-\sum _{j=1}^J{\widehat{\omega }}_j{Y}_{j,T}=A+B+C+D,$$

where

$$\begin{array}{cc}\hfill A& =\sum _{j=1}^J\left({\omega }_{\ast j}-{\widehat{\omega }}_j\right){\alpha }_j\hfill \\ \hfill B& =\underset{B\left(i\right)}{\underbrace{{\theta }_T\left(1-\sum _{j=1}^J{\omega }_{\ast j}\right)}}+\underset{B\left(ii\right)}{\underbrace{{\theta }_T\sum _{j=1}^J\left({\omega }_{\ast j}-{\widehat{\omega }}_j\right)}}\hfill \\ \hfill C& =\underset{C\left(i\right)}{\underbrace{{\left({\gamma }_0-\sum _{j=1}^J{\omega }_{\ast j}{\gamma }_j\right)}^{\prime }{\delta }_T}}+\underset{C\left(ii\right)}{\underbrace{{\left(\sum _{j=1}^J\left({\omega }_{\ast j}-{\widehat{\omega }}_j\right){\gamma }_j\right)}^{\prime }{\delta }_T}}\hfill \\ \hfill D& =\underset{D\left(i\right)}{\underbrace{\left({ϵ}_{0,T}-\sum _{j=1}^J{\omega }_{\ast j}{ϵ}_{j,T}\right)}}+\underset{D\left(ii\right)}{\underbrace{\left(\sum _{j=1}^J\left({\omega }_{\ast j}-{\widehat{\omega }}_j\right){ϵ}_{j,T}\right)}}.\hfill \end{array}$$

Note that the term B(i) is equal to 0 by design here, although it can be in principle different from 0 depending on the DGP and the estimator chosen. We speculate that the placebo test is used in the hope that (a) $Y_{0,T}-{\sum }_{j=1}^J{\widehat{\omega }}_j{Y}_{j,T}$ is dominated by the term D(i) above; (b) the four terms A, B(ii), C(ii) and D(ii) above, which reflect the noise of estimating ${\omega }_{\ast }$ by $\widehat{\omega }$, are ignorable; and (c) the two terms C(i) and D(i) more or less satisfy the symmetry property.

We argued in the previous section that the term D(i) is likely to violate the symmetry property. In order to assess the impacts of other terms, we consider the following variations in DGPs:

• Vary the values of $\alpha$’s such that (a) none of the components of ${\omega }_{\ast }$ dominates; (b) only two of the elements are non-zero.
• Vary the values of $\gamma$’s such that the unbalanced unobservable factors C(i) (a) disappear; and (b) are present.
• Vary $T_0$ such that the estimation errors in the weights are (a) prominent; and (b) negligible.

Combinations of the first two variations give us four different DGPs, shown as DGP No. 1 to No. 4 in Table 1.

Table 1. Data Generating Processes (DGPs) that generate size distortion.

DGP No.	$\mathit{\alpha }$’s	$\mathit{\gamma }$’s	Variations
1	${\alpha }_0=(2J+1)/3,{\alpha }_1=1,\cdots ,{\alpha }_J=J$	${\gamma }_0={\gamma }_1=\cdots ={\gamma }_J=0$	1(a), 2(a)
2	${\alpha }_0=(2J+1)/3,{\alpha }_1=1,\cdots ,{\alpha }_J=J$	${\gamma }_0=2,{\gamma }_1=\cdots ={\gamma }_J=1$	1(a), 2(b)
3	${\alpha }_0=5/3,{\alpha }_1=1,{\alpha }_2=2,{\alpha }_3=\cdots ={\alpha }_J=0$	${\gamma }_0={\gamma }_1=\cdots ={\gamma }_J=0$	1(b), 2(a)
4	${\alpha }_0=5/3,{\alpha }_1=1,{\alpha }_2=2,{\alpha }_3=\cdots ={\alpha }_J=0$	${\gamma }_0=2,{\gamma }_1=\cdots ={\gamma }_J=1$	1(b), 2(b)

We considered two versions of the placebo tests: the first one is what might be called a feasible version of the test. Formally, for $j=0,1,\dots ,J$, let $Y_j$ be a $T\times 1$ vector of outcomes for the jth control unit, let $Y=(Y_1,\dots ,Y_J)$, and let $Y_{-j}$ be a $T\times (J-1)$ matrix that deletes the jth column from Y. Then,

$$S(Y_0,Y)\equiv Y_{0,T}-\sum _{j=1}^J{\widehat{\omega }}_{\ast j}{Y}_{j,T}.$$

(7)

Similar to Equation (6), define the leave-one-out synthetic control weights ${\widehat{\omega }}_{-j}$ for the jth control unit as a solution to

$$min{\omega }^{\prime }\omega \mathrm{s}.\mathrm{t}.(\mathrm{i}){\overline{Y}}_{-j}^{\prime }\omega ={\overline{Y}}_j,(\mathrm{ii}){\ell }^{\prime }\omega =1,$$

(8)

where ${\overline{Y}}_{-j}$ is to delete the jth element from $\overline{Y}$. We likewise define the population counterpart ${\omega }_{\ast -j}$ as a solution to

$$min{\omega }^{\prime }\omega \mathrm{s}.\mathrm{t}.(\mathrm{i}){\alpha }_{-j}^{\prime }\omega ={\alpha }_j,(\mathrm{ii}){\ell }^{\prime }\omega =1.$$

For $j=1,\dots ,J$ and $k\ne j$, let ${\widehat{\omega }}_{-j,k}$ be the element in ${\widehat{\omega }}_{-j}$ that corresponds to the kth control unit. In addition, define ${\widehat{\omega }}_{-j,j}\equiv 0$ for $j=1,\dots ,J$. Then, for $j=1,\dots ,J$, we can compute

$$S\left(Y_j,Y_{-j}\right)\equiv Y_{j,T}-\sum _{k=1}^J{\widehat{\omega }}_{-j,k}{Y}_{k,T}.$$

(9)

Let $S^{\left(1\right)},\dots ,S^{\left(J\right)}$ be the order statistics of $S(Y_j,Y_{-j})$’s. We reject $H_0$ if $S(Y_0,Y)>S^{\left(J\left(1-\frac{\alpha }{2}\right)\right)}$ or $S(Y_0,Y)<S^{\left(\frac{J\alpha }{2}\right)}$.

The second test is an infeasible version of the test, which is identical to the first test, except that we use the true value of ${\omega }_{\ast }$, i.e.,

$$\begin{array}{cc}\hfill S^{true}(Y_0,Y)& \equiv Y_{0,T}-\sum _{j=1}^J{\omega }_{\ast j}{Y}_{j,T},\hfill \\ \hfill S^{true}(Y_j,Y_{-j})& \equiv Y_{j,T}-\sum _{k\ne j}^J{\omega }_{\ast -j,k}{Y}_{k,T},\hfill \end{array}$$

and we reject $H_0$ if $S^{true}(Y_0,Y)>S^{true,\left(J\left(1-\frac{\alpha }{2}\right)\right)}$ or $S^{true}(Y_0,Y)<S^{true,\left(\frac{J\alpha }{2}\right)}$.

For each DGP, we try $T_0\in \{40,80,400,800\}$, $J\in \{20,40,80\}$ and ${\sigma }_{ϵ}^2=0.1$. For all designs, we set the level of the tests to be $\alpha =10\%$, and the number of Monte Carlo runs to be 1000.

The results are summarized in Table 2.10 We see size distortions in Table 2, especially DGP No. 2 and No. 4. The size distortion there cannot be attributed to the noise of estimating $\omega$. First, the problem persists even as $T_0$ approaches unrealistically large values. Second, the size distortion is similar over the feasible and infeasible versions of the test. We suspect that the problem is a fundamental problem that may have something to do with the violation of symmetry. (An anonymous referee pointed out that DGPs No. 2 and No. 4 cannot produce synthetic controls that approximate the trajectory of the outcome for the treated, and that synthetic controls should not be applied in those settings.)

Table 2. Null rejection rates of permutation tests.

DGP	J	${\mathit{\sigma }}_{\mathit{\delta }}^2$	Estimated Weights				True Weights
			${\mathit{T}}_0=40$	${\mathit{T}}_0=80$	${\mathit{T}}_0=400$	${\mathit{T}}_0=800$	${\mathit{T}}_0=40$	${\mathit{T}}_0=80$	${\mathit{T}}_0=400$	${\mathit{T}}_0=800$
1		0.1	0.076	0.067	0.082	0.090	0.074	0.070	0.079	0.091
	$J=20$	1	0.076	0.067	0.082	0.090	0.074	0.070	0.079	0.091
		10	0.076	0.067	0.082	0.090	0.074	0.070	0.079	0.091
		0.1	0.079	0.080	0.080	0.093	0.079	0.090	0.084	0.096
	$J=40$	1	0.079	0.080	0.080	0.093	0.079	0.090	0.084	0.096
		10	0.079	0.080	0.080	0.093	0.079	0.090	0.084	0.096
		0.1	0.100	0.087	0.113	0.083	0.094	0.087	0.114	0.085
	$J=80$	1	0.100	0.087	0.113	0.083	0.094	0.087	0.114	0.085
		10	0.100	0.087	0.113	0.083	0.094	0.087	0.114	0.085
2		0.1	0.193	0.178	0.175	0.190	0.174	0.176	0.174	0.190
	$J=20$	1	0.568	0.530	0.559	0.529	0.557	0.518	0.561	0.531
		10	0.838	0.843	0.837	0.837	0.839	0.828	0.838	0.838
		0.1	0.193	0.219	0.198	0.208	0.200	0.213	0.193	0.207
	$J=40$	1	0.561	0.623	0.586	0.589	0.556	0.593	0.588	0.583
		10	0.809	0.853	0.863	0.844	0.832	0.858	0.854	0.846
		0.1	0.238	0.253	0.245	0.230	0.239	0.245	0.241	0.233
	$J=80$	1	0.617	0.623	0.631	0.619	0.610	0.629	0.624	0.622
		10	0.865	0.870	0.870	0.866	0.861	0.894	0.867	0.868
3		0.1	0.115	0.123	0.110	0.125	0.114	0.113	0.109	0.123
	$J=20$	1	0.115	0.123	0.110	0.125	0.114	0.113	0.109	0.123
		10	0.115	0.123	0.110	0.125	0.114	0.113	0.109	0.123
		0.1	0.152	0.151	0.149	0.153	0.153	0.154	0.146	0.154
	$J=40$	1	0.152	0.151	0.149	0.153	0.153	0.154	0.146	0.154
		10	0.152	0.151	0.149	0.153	0.153	0.154	0.146	0.154
		0.1	0.180	0.168	0.173	0.153	0.178	0.164	0.166	0.153
	$J=80$	1	0.180	0.168	0.173	0.153	0.178	0.164	0.166	0.153
		10	0.180	0.168	0.173	0.153	0.178	0.164	0.166	0.153
4		0.1	0.201	0.219	0.193	0.214	0.207	0.213	0.195	0.209
	$J=20$	1	0.536	0.518	0.525	0.522	0.533	0.520	0.525	0.520
		10	0.837	0.827	0.817	0.814	0.841	0.809	0.824	0.812
		0.1	0.238	0.253	0.249	0.253	0.241	0.268	0.239	0.249
	$J=40$	1	0.549	0.597	0.583	0.579	0.552	0.576	0.583	0.582
		10	0.809	0.842	0.856	0.852	0.828	0.843	0.859	0.848
		0.1	0.287	0.293	0.300	0.275	0.279	0.301	0.297	0.278
	$J=80$	1	0.610	0.623	0.655	0.629	0.608	0.625	0.653	0.634
		10	0.861	0.871	0.874	0.860	0.866	0.880	0.865	0.866

Our Monte Carlo analysis indicates that the placebo test does have the size distortion problem. The results in Table 2 suggest that the size problem is potentially bigger in DGPs No. 2 and No. 4. DGPs No. 2 and No. 4 differ from No. 1 and No. 3 in that the $\gamma$’s are nonzero and the aggregate shock ${\delta }_t$ plays a role as a consequence. Therefore, it is of interest to investigate further sources of asymmetry. For this purpose, we revisit the decomposition in Equation (5) of $Y_{0,T_0+1}\left(0\right)-{\sum }_{j=1}^J{\omega }_{\ast j}{Y}_{j,T_0+1}\left(0\right)$, assuming that the first and second terms in the factor model in Equation (3) are not present:11

$$Y_{0,T_0+1}\left(0\right)-\sum _{j=1}^J{\omega }_{\ast j}{Y}_{j,T_0+1}\left(0\right)={\left({\gamma }_0-\sum _{j=1}^J{\omega }_{\ast j}{\gamma }_j\right)}^{\prime }{\delta }_{T_0+1}+\left({ϵ}_{0,T_0+1}-\sum _{j=1}^J{\omega }_{\ast j}{ϵ}_{j,T_0+1}\right).$$

This implies that the variance of $Y_{0,T_0+1}\left(0\right)-{\sum }_{j=1}^J{\omega }_{\ast j}{Y}_{j,T_0+1}\left(0\right)$ can be written as

$${\left({\gamma }_0-\sum _{j=1}^J{\omega }_{\ast j}{\gamma }_j\right)}^{\prime }\mathsf{\Sigma }\left({\delta }_t\right)\left({\gamma }_0-\sum _{j=1}^J{\omega }_{\ast j}{\gamma }_j\right)+\mathrm{var}\left({ϵ}_{j,t}\right)\left(1+\sum _{j=1}^J{\omega }_{\ast j}^2\right)$$

under the assumptions of the DGPs, where $\mathsf{\Sigma }\left({\delta }_t\right)$ is the covariance matrix of the vector ${\delta }_t$. Likewise, the variances of the permutation statistics are

$${\left({\gamma }_j-\sum _{k\ne j}{\omega }_{\ast -j,k}{\gamma }_k\right)}^{\prime }\mathsf{\Sigma }\left({\delta }_t\right)\left({\gamma }_j-\sum _{k\ne j}{\omega }_{\ast -j,k}{\gamma }_k\right)+\mathrm{var}\left({ϵ}_{j,t}\right)\left(1+\sum _{k\ne j}{\omega }_{\ast -j,k}^2\right).$$

Depending on the relative magnitudes of $\gamma$’s, we can easily construct examples that violate the symmetry, such as DGPs No. 2 and No. 4. As of now, it is not clear to us whether there is another venue (other than the variation in the size of $\gamma$), which leads to a violation of the symmetry.

4. Possible Alternatives to Placebo Tests

If we take the time series asymptotics ($T_0\to \infty$) seriously, the problem can be avoided by using the same idea as in Andrews (2003). The hypothesis of no treatment effects can be understood to be a hypothesis of stationarity of the time series $W_t\equiv Y_{0,t}-{\sum }_{j=1}^J{\omega }_{\ast j}{Y}_{j,t}$. In particular, the researcher is interested in whether the distribution of $W_{T_0+1},\dots ,W_T$ is the same as that of $W_1,\dots ,W_{T_0}$, for which Andrews (2003)’s test is well-suited. In the simple case that we consider where $T=T_0+1$, one rejects the null if $W_{T_0+1}$ belongs to the extreme tails of the empirical distribution of $W_1,\dots ,W_{T_0}$. We conducted Monte Carlo simulations for all the DGPs considered in the previous section, and verified that Andrews (2003)’s test suffered no size distortion.12 Andrews (2003)’s test is geared for application in time series, and as such, robust to certain heteroscedasticity. If the variances of ${ϵ}_{j,t}$ in Equation (3) were different across js, most of the available methods exploiting cross sectional variation may need to be used with caution, as noted by Ferman and Pinto (2017). Andrews (2003)’s end-of-sample instability test being a test of stationarity of $Y_{0,t}-{\sum }_{j=1}^J{\omega }_{\ast j}{Y}_{j,t}$, its validity does not depend on whether the ${ϵ}_{j,t}$’s have identical variances or not. The usefulness of Andrews (2003)’s test in this context was recognized earlier by Ferman and Pinto (2017).

Andrews (2003)’s test utilizes time series variation seriously. When $T_0$ is relatively small, perhaps the researcher would like to have a procedure that is based on cross sectional variation. If the factor structure is taken seriously and if the number of factors is a priori known, we can produce such a procedure by combining the ideas in Conley and Taber (2011) and Holtz-Eakin et al. (1988). For simplicity, assume that the model is given by

$$\begin{array}{cc}\hfill Y_{j,t}=& Y_{j,t}\left(0\right)={\gamma }_j{\delta }_t+x_{j,t}^{\prime }\beta +{ϵ}_{j,t}t=1,2;j=1,\dots ,J,\hfill \\ & Y_{0,t}\left(0\right)={\gamma }_0{\delta }_t+x_{0,t}^{\prime }\beta +{ϵ}_{0,t},\hfill \end{array}$$

where we normalize ${\delta }_1=1$. Let $Y_{0,1}=Y_{0,1}\left(0\right)$ and $Y_{0,2}=Y_{0,2}\left(1\right)=Y_{0,2}\left(0\right)+\eta$. This is a case where $T_0=1$, and $T=2$. We then have

$$\frac{Y_{j,2}}{{\delta }_2}-Y_{j,1}={\left(\frac{x_{j,2}}{{\delta }_2}-x_{j,1}\right)}^{\prime }\beta +\frac{{ϵ}_{j,2}}{{\delta }_2}-{ϵ}_{j,1}t=1,2;j=1,\dots ,J.$$

Under strict exogeneity assumption on x’s, we can consistently estimate $\left(\beta ,{\delta }_2\right)$ as $J\to \infty$ by using the control group. Now, assume that $\left({ϵ}_{j,1},{ϵ}_{j,2}\right)$ $j=0,1,2,\dots$ are i.i.d., which would imply

$$\frac{{ϵ}_{j,2}}{{\delta }_2}-{ϵ}_{j,1}=\frac{Y_{j,2}-x_{j,2}^{\prime }\beta }{{\delta }_2}-\left(Y_{j,1}-x_{j,1}^{\prime }\beta \right)$$

are i.i.d. A simple modification of Conley and Taber (2011)’s argument establishes that the distribution of $\frac{{ϵ}_{j,2}}{{\delta }_2}-{ϵ}_{j,1}$ can be consistently estimated by the empirical distribution of

$$\frac{Y_{j,2}-x_{j,2}^{\prime }\widehat{\beta }}{{\widehat{\delta }}_2}-\left(Y_{j,1}-x_{j,1}^{\prime }\widehat{\beta }\right)j=1,\dots ,J,$$

where $\left(\widehat{\beta },{\widehat{\delta }}_2\right)$ denotes Holtz-Eakin et al. (1988)’s estimator. Therefore, in order to test that $\eta =\overline{\eta }$, it suffices to consider a test that rejects whenever

$$\frac{Y_{0,2}-\overline{\eta }-x_{0,2}^{\prime }\widehat{\beta }}{{\widehat{\delta }}_2}-\left(Y_{0,1}-x_{0,1}^{\prime }\widehat{\beta }\right)$$

is in the extreme tails of such empirical distribution. Ahn et al. (2013), for example, discussed how Holtz-Eakin et al. (1988)’s method can be generalized when there are multiple factors. The idea of combining Holtz-Eakin et al. (1988) with Conley and Taber (2011), although straightforward, does not seem to have been considered elsewhere.

We have considered two alternative methods of inference, one based on $T_0\to \infty$ asymptotics, and the other based on $J\to \infty$ asymptotics. In addition to these two methods, we can also entertain the possibility that if both $T_0$ and J are large, it may be possible to use the panel technique as in Bai (2009) as well.13 See, e.g., Gobillon and Magnac (2016). The latter two procedures are based on the presumption that the researcher takes the linear factor structure seriously, so it may be more powerful than the Andrews (2003)’s test. On the other hand, if a researcher views the linear factor model as just a toy model14 to illustrate the potential problem of difference-in-differences methods, then she would probably be hesitant to discard the synthetic control method, which may be able to accommodate potentially complicated statistical structures that may go beyond the linear factor model.

The three methods that we discussed here as possible alternatives are all theoretically valid under some asymptotics. Asymptotic validity does not necessarily imply that any given method performs reasonably for a given finite sample. A serious Monte Carlo comparison of the relative performance of the three alternatives, which is beyond the scope of the current paper, is required to determine a method to be recommended to practitioners.

5. Conclusions

We considered the performance of the permutation test (placebo test) in the context of the synthetic control method. The symmetry assumption, one of the crucial conditions for the validity of the permutation test, may be violated in synthetic control studies. Using Monte Carlo simulations, we show that the size of the permutation tests can be distorted. The results suggest that even with simple DGPs and rather restrictive distributional assumptions of the error term, as long as aggregate shocks are present, the permutation test in its current form is likely to fail and cannot serve as a proper tool for inference with the synthetic control method. Several possible alternatives were discussed. That being said, we should be careful and repeat an anonymous referee’s cautious remark that, while our analysis is from a repeated sampling perspective, the synthetic control literature uses permutation tests in the context of design-based inference, and as such, the permutation tests have exact size.