# Full text of "ERIC ED512680: Compliance-Effect Correlation Bias in Instrumental Variables Estimators"

## See other formats

Abstract Title Page Not included in page count. Title: Compliance-Effect Correlation Bias in Instrumental Variables Estimators Author(s): seanf. reardon 2010 SREE Conference Abstract Template Abstract Body Background/context: Instrumental variables estimators hold the promise of enabling researchers to estimate the effects of educational treatments that are not (or cannot be) randomly assigned but that may be affected by randomly assigned interventions. Examples of the use of instrumental variables in such cases are increasingly common in educational and social science research. For example, although we cannot randomly assign students to attend private school, we can use the variation in private school enrollment induced by the (randomly-assigned) offer of a tuition voucher to identify and estimate the effect of private school enrollment (Krueger and Zhu 2004). Although we cannot randomly assign teachers to use specific instructional practices, we can use the variation in instructional practices induced by a (randomly-assigned) professional development program to identify and estimate the effect of specific instructional practices. The most commonly used instrumental variables estimator is two-stage least squares (2SLS). Many of the properties of the 2SLS estimator are well-understood, including its identifying assumptions (Angrist, Imbens and Rubin 1996) and the issue of finite sample bias (Bound, Jaeger and Baker 1995). Less well understood, however, is the behavior of the 2SLS estimator when individuals are heterogeneous in their compliance with the instrument (that is, heterogeneous in the extent to which the instrument affects their behavior). Heckman, Urzua, and Vytlacil (2006), for example, note a peculiar asymmetry in the assumptions of many instrumental variables models: they typically allow heterogeneity of compliance, but assume homogeneity of treatment effects. This paper investigates the effects of heterogeneous compliance on 2SLS estimates and the conditions under which such heterogeneous compliance results in bias. A limitation of instrumental variables models is that they are able to identify the effects of only (at most) as many mediators as there are ignorably assigned instruments. Thus, if we think that a specific intervention may affect student outcomes through several different mediators, we require at least as many instruments as mediators. Multi-site randomized trials can be used to define a set of instruments, each defined as the interaction of treatment status with a site indicator variable. If there are more sites than mediators, then such multisite trials hold the promise of enabling researchers to estimate the effects of multiple mediators at once. The issue of heterogeneous compliance, however, is particularly important in the case where site-by-treatment interactions are included as instruments, for in this case, it is precisely the heterogeneity of the compliance across sites that is being leveraged to provide identification of effects. In fact, if sites are homogeneous with respect to their compliance, the inclusion of site- by-treatment interactions as instruments leads to finite sample bias because the combined set of instruments is much weaker in this case than a single instrument would be. Purpose / objective / research question / focus of study: The above discussion suggests that the use of instrumental variables estimators to identify the effects of mediators in multi-site randomized trials requires a thorough understanding of the effects of compliance heterogeneity and the conditions under which it may lead to bias. In this 2010 SREE Conference Abstract Template A-l paper, I investigate the properties of the 2SLS IV estimator under conditions of heterogeneous compliance. I do so in both the simple case where a 2SLS model is used to estimate the effect of single mediator in a single site, and the the more complex case where multiple site-by- treatment interactions are used to estimate the effects of multiple mediators. Setting: N/A: methodological paper Population / Participants / Subjects: N/A: methodological paper Intervention / Program / Practice: N/A: methodological paper A Stylized Example: To fix ideas, suppose we are interested in investigating how a teacher professional development (PD) program (call it T) affects student achievement (call it Y). Further, suppose the theory of action of the PD program is that it will affect teachers' use of three instructional practices (call them A, B, and C), which in turn will affect student achievement. We say that T acts on Y through the mediators A, B, and C. We therefore have three research questions: 1) what is the average effect of the PD program on student achievement? 2) what is the average effect of the PD program on teachers' instructional practices? and 3) what are the average effects of each of the instructional practices A, B, and C on student achievement? Suppose we design a study in which we sample a moderate number of school districts, and in each district we randomize schools to receive T or not. In each school, we observe teachers' instructional practices and measure student achievement at the end of some time period. At the end of the time period, we can easily obtain an unbiased estimate the average effect of the PD program T on student achievement (because T is randomly assigned). Moreover, we can likewise obtain an unbiased estimate the average effect of the PD program T on teachers' instructional practices (again, because T is randomly assigned). Obtaining an unbiased estimate of the effect of instructional practices A, B, and C on student achievement, however, is more complex, because teachers' instructional practices are not randomly assigned in this design. In cases like this, researchers sometimes rely on instrumental variables (IV) methods to obtain estimates of the effects of mediators on outcomes. Under the assumption that the treatment T can only affect the outcome Y through its effects on mediators A, B, and C (as well as some additional assumptions), we may be able to identify the effects of the mediators themselves by relying on the associations among the treatment-induced variation in the mediators and the treatment-induced variation in the outcome. If districts where the PD program had larger average effects on instructional practice A are also those where the PD program had larger average effects on student achievement, controlling for the average effects of the PD program on instructional practices B and C, then we may be able to conclude that A affects Y. 2010 SREE Conference Abstract Template A-2 Research Design: N/A: methodological paper Data Collection and Analysis: N/A: methodological paper Notation and Preliminaries: Let there be N individuals, indexed by i, who are located in S sites, indexed by s. Let there be P mediators M 1 , M 2 , ... M p , indexed by p (and sometimes indexed by q). Each individual is assigned a treatment T is (we assume T is ignorably assigned within each site) and we observe mediator values m v is (which may be affected by T) and outcome Y is for each individual. In what follows, we will assume each of the five IV assumptions described by Angrist, Imbens, and Rubin hold (that is, we assume SUTVA; we assume a non-zero effect of T on each M p ; we assume monotonicity; ignorable treatment assignment; and the exclusion restriction) (Angrist, Imbens and Rubin 1996). Further, we will assume that the samples are large enough, and the treatment-induced variation in the mediators is strong enough, that finite sample bias is inconsequential (Bound, Jaeger and Baker 1995). We assume that the structural model describing the relationship between the outcome Y and the mediators M p is of the form: Yis = Ac + r Yj 5Pm is + It; p = 1 where A s is a site-specific fixed effect (we need this because we have assumed ignorable treatment assignment conditional on sites), u is is an iid mean-zero, normally distributed error, and 8 P is the average effect of M p on Y in the sampled population. By the exclusion restriction, T does not appear in the structural model. We are interested in estimating each of the 8 p 's. The goal of this paper is to specify the conditions under which 2SLS will yield an unbiased estimate of each of the 8 p 's. We are particularly interested in knowing whether the § p estimated via 2SLS is equal to the average of the 5/s. The paper proceeds by deriving the estimand of the 2SLS estimator in several cases. I begin with the simple case when there is one site and one mediator. I then consider cases where treatment is randomly assigned within multiple sites but there is only a single mediator. Finally, I consider the most general case, where there are multiple sites and multiple mediators. Findings / Results: Case 1: S-l, P-1 : Consider the system of equations mj =7% + ef Y t = 5 x m^ + itj 2010 SREE Conference Abstract Template A-3 Here, y 1 is the average effect of T on mediator m 1 (that is, it is the average of the person- specific compliance, y }), and 5 1 is the average effect of mediator m 1 on outcome Y (that is, it is the average of the person-specific effect, Sf). In the paper, I show that, in infinite samples, 2SLS estimates converge to the estimand p/im[5 1(2SiS) ] Si Cov(yl,Sl ) = 6 -I- y 1 That is, the 2SLS estimand is the compliance-weighted average treatment effect (as shown in the first line), not the simple average treatment effect we desire. The second line writes this same estimand as the average treatment effect plus a bias term, which will be non-zero if the person-specific compliance (the effect of T on M 1 ) is correlated with the person-specific effect of M 1 on Y among individuals (I refer to this as the compliance-effect correlation). The compliance-effect correlation bias will be exacerbated when T is a weak instrument (when y 1 is small). Case 2: S>1, P-1 : In this case, we have multiple sites, but only a single mediator. A standard approach is to use the multiple site-by-treatment interactions to generate S instruments (see, for example, Katz, Kling and Liebman 2007). We then have the following pair of equations: s m is = r s + ^ YrOis ■ T is ) + els r = 1 Yi s A s + 5 zn.j s + In the paper, I show that estimating this model via 2SLS yields S n s (jKYs ) 2 Ss^s^sCys 1 ) 2 Cov s (yf s , Ys 1 ) where n s is the number of sampled individuals in site s; of is the variance of the treatment in site s; yf is the average compliance within site s; 5f is the average effect of M 1 in site s, and Cov s (yi s , Sif) is the compliance-effect covariance within site s. This result indicates that there are two sources of bias in the 2SLS estimator: 1) one due to the correlation between the square of the site-average compliance ((/s 1 ) 2 ) an d the site-average effect ((*> 5 ) (I call this the between-site compliance-effect bias;); and 2) one due to the average within-site correlations between person-specific compliance and person-specific effects (I call this the within-site compliance-effect bias). 2010 SREE Conference Abstract Template A-4 Case 3: S>P>1 : In this case, we have multiple sites, and multiple mediators (but more sites than mediators in order that the model be identified). A standard approach is to use the multiple site-by-treatment interactions to generate S instruments (see, for example, Morris, Duncan and Rodrigues 2006). We then have the following system of equations: This implies the system of equations: s m is = f s + ^ Yr 01s ■ T is ) + 4 r= 1 S Y YrOIs ■ T is ) + 4 mi = It + Tis — d" I r=l Yr Ois ' T is ) + 4 y S p m v is + itj, v In the paper, I show that fitting this set of equations via 2SLS yields the following: plim[S p ( 2SLS )] = y S 1^-sOsYI Cov s(y? S ’ 5 1) \ Ys ) where a pq is a weight term (it is the element in the p th row and q th column of the inverse of the weighted variance-covariance average compliance matrix). This shows that the 2SLS estimator converges to an estimand that is a complex weighted average of the person-specific effects in the case of a multi-site multiple-mediator estimator. There are two kinds of bias here: within-site compliance-effect correlation bias; and bias due to a correlation between the site-specific weights and the site average treatment effects. The weights are complex and relatively uninterpretable, suggesting that the 2SLS estimator in this case does not in general yield a meaningful estimand. Conclusions: The paper shows that the 2SLS estimator may be severely biased by correlations between instrument compliance and mediator effects, even if all the standard IV assumptions are met. This bias is in addition to finite sample bias, and is, in fact, exacerbated by weak instruments. Because there are many conditions that might yield compliance-effect correlations, this source of bias should be considered in designing and interpreting the results of studies that rely on instrumental variables estimation. 2010 SREE Conference Abstract Template A-5 Appendices Appendix A. References References are to be in APA version 6 format. Angrist, Joshua D., Guido W. Imbens, and Donald B. Rubin. 1996. "Identification of Causal Effects Using Instrumental Variables." Journal of the American Statistical Association 91:444-455. Bound, John, David A. Jaeger, and Regina M. Baker. 1995. "Problems with Instrumental Variables Estimation When the Correlation Between the Instruments and the Endogenous Explanatory Variable is Weak." Journal of the American Statistical Association 90:443-450. Heckman, James J., Sergio Urzua, and Edward Vytlacil. 2006. "Understanding instrumental variables in models with essential heterogeneity." Review of Economics and Statistics 88:389-432. Katz, Lawrence F., Jeffrey R. Kling, and Jeffrey B. Liebman. 2007. "Experimental estimates of neighborhood effects." Econometrica 75:83-119. Krueger, Alan B., and Pei Zhu. 2004. "Another Look at the New York City Voucher Experiment." American Behavioral Scientist 47:658-698. Morris, Pamela, Greg J. Duncan, and Christopher Rodrigues. 2006. "Does Money Really Matter? Estimating Impacts of Family Income on Young Children's Achievement with Data from Random-Assignment Experiments." Unpublished manuscript. 2010 SREE Conference Abstract Template A-6 Appendix B. Tables and Figures Not included in page count. 2010 SREE Conference Abstract Template B-l