Clustered observational studies (COSs) are a critical analytic tool for educational effectiveness research. We present a design framework for the development and critique of COSs. The framework is built on the counterfactual model for causal inference and promotes the concept of designing COSs that emulate the targeted randomized trial that would have been conducted were it feasible. We emphasize the key role of understanding the assignment mechanism to study design. We review methods for statistical adjustment and highlight a recently developed form of matching designed specifically for COSs. We review how regression models can be profitably combined with matching and note best practice for estimates of statistical uncertainty. Finally, we review how sensitivity analyses can determine whether conclusions are sensitive to bias from potential unobserved confounders. We demonstrate concepts with an evaluation of a summer school reading intervention in Wake County, North Carolina.
Many interventions in education occur in settings where treatments are applied to groups. For example, a reading intervention may be implemented for all students in some schools and withheld from students in other schools. When such treatments are non-randomly allocated, outcomes across the treated and control groups may differ due to the treatment or due to baseline differences between groups. When this is the case, researchers can use statistical adjustment to make treated and control groups similar in terms of observed characteristics. Recent work in statistics has developed matching methods designed for contexts where treatments are clustered. This form of matching, known as multilevel matching, may be well suited to many education applications where treatments are assigned to schools. In this article, we provide an extensive evaluation of multilevel matching and compare it to multilevel regression modeling. We evaluate multilevel matching methods in two ways. First, we use these matching methods to recover treatment effect estimates from three clustered randomized trials using a within-study comparison design. Second, we conduct a simulation study. We find evidence that generally favors an analytic approach to statistical adjustment that combines multilevel matching with regression adjustment. We conclude with an empirical application.
This paper presents final results for a two-year randomized controlled trial of multi-tiered system of supports (MTSS) in the Wake County Public School System. MTSS is a district-wide comprehensive school reform model designed to improve academic outcomes and reduce behavioral incidents through the delivery of tiered supports. MTSS was randomly assigned within 44 school-pairs in fall 2015. Treatment and control groups were balanced along student-level characteristics, prior achievement, and prior behavioral incidents. After the second year of implementation, MTSS did not significantly impact math achievement but did have an empirically small impact on elementary school reading. Hispanic students had the largest gains in reading and math at the elementary and middle school levels. MTSS did not significantly impact high school outcomes and had largely negative impacts on certain subgroups. The intervention did not impact short-term suspension counts and led to an unexpected increase in suspensions among select subgroups. MTSS did not meaningfully impact rates of special education referral or downstream outcomes such as high school graduation and dropout. The results herein suggest that the district’s random assignment of MTSS to a large sample of schools across grade levels did not have the hypothesized effects on achievement or behavior.
This paper examines curricular acceleration in mathematics during elementary school using administrative data from a large, diverse school district that recently implemented a targeted, test-based acceleration policy. We first characterize access to advanced math and then estimate effects of acceleration in math on measures of short-run academic achievement as well as non- test-score measures of grit, engagement with schoolwork, future plans, and continued participation in the accelerated track. Experiences and effects of math acceleration differ markedly for girls and boys. Girls are less likely to be nominated for math acceleration and perform worse on the qualifying test, relative to boys with equivalent baseline performance. We find negative effects of acceleration on short-run retention of math knowledge for girls, but no such performance decay for boys. After initial exposure to accelerated math, girls are less likely than boys to appear in the accelerated track during late elementary school and at the start of middle school.
The American Academy of Pediatrics recommends high schools begin after 8:30 AM to better align with the circadian rhythms of adolescents. Yet due to economic and logistic considerations such as transportation, athletics, and students’ after-school employment, the vast majority of high schools begin the school day considerably earlier. We leverage a quasi-natural experiment whereby five comprehensive high schools in a large and diverse school district moved start times forty minutes earlier to better coordinate with high schools already starting at 7:25 AM. In this setting, disruption effects from moving start times should exacerbate any harmful consequences of earlier start times. Early start times might negatively impact test scores, student engagement, and non-cognitive skill formation. We report on the effect of earlier start times on a broad range of outcomes, including mandatory ACT test scores, absenteeism, on-time progress in high school, and college-going. While we fail to find evidence of harmful effects on test scores, we do see a rise in absenteeism and tardiness rates, as well as higher rates of dropping out of high school. These results suggest that the harmful effects of early start times may not be well captured by considering test scores alone.
In the wake of legal challenges facing race-based integration, districts have turned tosocioeconomic integration in an attempt to achieve greater racial balance. Empirically, the extentto which these initiatives generate such balance is an open question. In this paper, we leveragethe school assignment system that the Wake County Public School System (WCPSS) employedto provide evidence on this issue. Although our results show that WCPSS’ socioeconomic-basedassignment policy had negligible effects on average levels of segregation across the district, itsubstantially reduced segregation for students who would have attended highly segregatedschools under a residence-based assignment policy. The policy also exposed these students topeers with different racial/ethnic backgrounds, higher achievement levels, and more advantagedneighborhood contexts.
Career academies serve an increasingly wide range of students. This paper examines the contemporary profile of students entering career academies in a large, diverse school district and estimates causal effects of participation in one of the district's well-regarded academies on a range of high school and college outcomes. Exploiting the lottery-based admissions process of this technology-focused academy, we find that academy enrollment increases the likelihood of high school graduation by about 8 percentage points and boosts rates of college enrollment for males but not females. Analysis of intermediate outcomes suggests that effects on attendance and industry-relevant certification at least partially mediate the overall high school graduation effect.
This article explores the origins of youth engagement in school, community and democracy. Specifically, it considers the role of psychosocial or non-cognitive abilities, like grit or perseverance. Using a novel original large-scale longitudinal survey of students linked to school administrative records and a variety of modeling techniques – including sibling, twin and individual fixed effects – the study finds that psychosocial abilities are a strong predictor of youth civic engagement. Gritty students miss less class time and are more engaged in their schools, are more politically efficacious, are more likely to intend to vote when they become eligible, and volunteer more. Our work highlights the value of psychosocial attributes in the political socialization of young people.
There are sizable and pervasive academic achievement gaps between minority and non-minority students in the United States. Non-minority students – particularly boys – are more likely to enroll in school one year after they become eligible, a practice known as ‘redshirting.’ Consequently, non-minority students are on average more mature than minority students when they take standardized tests. Many studies have documented that differences in maturity at the moment of testing translate into large differences in test scores. Thus, differences in redshirting behavior across minority and non-minority students may be a contributing factor to achievement gaps. This study analyzes the effect of redshirting on achievement gaps using a reform in North Carolina that shifted the cutoff date for school eligibility in 2009 from October 16 to August 31. We use the reform to create an instrumental variable for redshirting behavior. Using data for eight cohorts of 3rd graders in the Wake County Public School System and a difference-in-differences approach, we estimate that redshirting increases the achievement gap by 28%–30% among boys born close to the cutoff date for school eligibility, and 3%–4% among all boys. For girls, the estimates are 8%–11% for those born close to the cutoff and 1% overall, but these estimates lack statistical significance. We discuss some policy implications of shifting the cutoff date for school eligibility – 14 states have done since 2000 – and growing redshirting rates.
Many observational studies of causal effects occur in settings with clustered treatment assignment. In studies of this type, treatment is applied to entire clusters of units. For example, an educational in- tervention might be administered to all the students in a school. We develop a matching algorithm for multilevel data based on a network flow algorithm. Earlier work on multilevel matching relied on integer programming, which allows for balance targeting on specific covari- ates but can be slow with larger data sets. Although we cannot di- rectly specify minimal levels of balance for individual covariates, our algorithm is fast and scales easily to larger data sets. We apply this algorithm to assess a school-based intervention through which stu- dents in treated schools were exposed to a new reading program dur- ing summer school. In one variant of the algorithm, where we match both schools and students, we change the causal estimand through optimal subset matching to better maintain common support. In a second variant, we relax the common support assumption to preserve the causal estimand by only matching on schools. We find that the summer intervention does not appear to increase reading test scores. In a sensitivity analysis, however, we determine that an unobserved confounder could easily mask a larger treatment effect.
All papers listed in the Submitted and Working Papers sections have complete paper drafts. Please email me if you would like to see the most recent draft.