Publications by Type: Journal Article

Working Paper
D.V. Hill, M.A. Lenard, and L.C. Page. Working Paper. “Differentiated Literacy and Student Achievement: Evidence from a Randomized Controlled Trial of Achieve3000”.
Submitted
U. Dur, B. Hammond, M.A. Lenard, M. Morrill, T. Morrill, and C. Paeplow. Submitted. “The Attraction of Magnet Schools: Evidence from Embedded Lotteries in School Assignment”. Working PaperAbstract
Magnet schools aim to diversify enrollment by providing innovative curricula to attract students from other schools within a school district.  Measuring the impact of attending a magnet is challenging because students choose to apply and schools have priorities over types of students. This study estimates the causal impact of attending a magnet school by leveraging exogenous variation arising from tie breakers embedded in a centralized school assignment mechanism. Attending a magnet led to higher performance in mathematics, and non-language immersion magnet schools also increased students' reading scores.  Magnet schools also significantly increase student engagement, suggesting improved non-cognitive skill formation.
M.A. Lenard and M. Silliman. Submitted. “Informal social interactions, academic achievement, and behavior”. EdWorkingPapersAbstract
We study the effects of informal social interactions on academic achievement and behavior using idiosyncratic variation in peer groups stemming from changes in bus routes across elementary, middle, and high school. In early grades, a one standard-deviation change in the value-added of same-grade bus peers corresponds to a 0.01 SD change in academic performance and a 0.03 SD change in behavior; by high school, these magnitudes grow to 0.04 SD and 0.06 SD. These findings suggest that student interactions outside the classroom—especially in adolescence—may be an important factor in the education production function.
2023
D.V. Hill, R. P. Hughes, M.A. Lenard, D.D. Liebowitz, and L.C. Page. 2023. “New Schools and New Classmates: The Disruption and Peer Group Effects of School Reassignment.” Economics of Education Review, 92, 102316. Publisher's VersionAbstract
Policy makers periodically consider using student assignment policies to improve educational outcomes by altering the socio-economic and academic skill composition of schools. We exploit the quasi-random reassignment of students across schools in the Wake County Public School System to estimate the academic and behavioral effects of being reassigned to a different school and, separately, of shifts in peer characteristics. We rule out all but substantively small effects of transitioning to a different school as a result of reassignment on test scores, course grades and chronic absenteeism. In contrast, increasing the achievement levels of students’ peers improves students’ math and ELA test scores but harms their ELA course grades. Test score benefits accrue primarily to students from higher-income families, though students with lower family income or lower prior performance still benefit. Our results suggest that student assignment policies that relocate students to avoid the over-concentration of lower-achieving students or those from lower-income families can accomplish equity goals (despite important caveats), although these reassignments may reduce achievement for students from higher-income backgrounds.
2022
Olivia L. Chi and Matthew A. Lenard. 2022. “Can a Commercial Screening Tool Help Select Better Teachers?” Educational Evaluation and Policy Analysis. Publisher's VersionAbstract
Improving teacher selection is an important strategy for strengthening the quality of the teacher workforce. As districts adopt commercial teacher screening tools, evidence is needed to understand these tools’ predictive validity. We examine the relationship between Frontline Education’s TeacherFit instrument and newly hired teachers’ outcomes. We find that a 1 SD increase on an index of TeacherFit scores is associated with a 0.06 SD increase in evaluation scores. However, we also find evidence that teachers with higher TeacherFit scores are more likely to leave their hiring schools the following year. Our results suggest that TeacherFit is not necessarily a substitute for more rigorous screening processes that are conducted by human resources officials, such as those documented in recent studies.
L. Keele, M. Lenard, and L. Page. 2022. “Overlap Violations in Clustered Observational Studies of Educational Interventions.” Journal of Research on Educational Effectiveness. Publisher's VersionAbstract
In education settings, treatments are often non-randomly assigned to clusters, such as schools or classrooms, while outcomes are measured for students. This research design is called the clustered observational study (COS). We examine the consequences of common support violations in the COS context. Common support violations occur when the covariate distributions of treated and control units have substantial areas of non-overlap. Such violations are likely to occur in a COS, especially with a small number of treated clusters. We provide a comprehensive review of methods for overlap violations in the context of COS designs. We provide an overview of diagnostic tests and trimming methods to ensure overlap holds for the distributions of treated and control covariates. We then outline how trimming changes the estimand and how profiling can be used to understand the causal quantity for which overlap holds. Finally, we demonstrate how steps to achieve adequate overlap can result in very narrowly defined causal effects that may have little policy relevance. We use data on Catholic schools to illustrate concepts throughout.
2021
Thurston Domina, Deven Carlson, James S. Carter III, Matthew A. Lenard, Andrew McEachin, and Rachel Perera. 2021. “The Kids on the Bus: The Academic Consequences of Diversity-Driven School Reassignments.” Journal of Policy Analysis and Management. Publisher's VersionAbstract
Many public school diversity efforts rely on reassigning students from one school to another. While opponents of such efforts articulate concerns about the consequences of reassignments for students’ educational experiences, little evidence exists regarding these effects, particularly in contemporary policy contexts. Using an event study design, we leverage data from an innovative socioeconomic school desegregation plan to estimate the effects of reassignment on reassigned students’ achievement, attendance, and exposure to exclusionary discipline. Between 2000 and 2010, North Carolina’s Wake County Public School System (WCPSS) reassigned approximately 25 percent of students with the goal of creating socioeconomically diverse schools. Although WCPSS’s controlled school choice policy provided opportunities for reassigned students to opt out of their newly reassigned schools, our analysis indicates that reassigned students typically attended their newly reassigned schools. We find that reassignment modestly boosts reassigned students’ math achievement, reduces reassigned students’ rate of suspension, and has no offsetting negative consequences on other outcomes. Exploratory analyses suggest that the effects of reassignment do not meaningfully vary by student characteristics or school choice decisions. The results suggest that carefully designed school assignment policies can improve school diversity without imposing academic or disciplinary costs on reassigned students.
Luke Keele, Matthew A. Lenard, and Lindsay C. Page. 2021. “Matching Methods for Clustered Observational Studies in Education.” Journal of Research on Educational Effectiveness. Publisher's VersionAbstract
Many interventions in education occur in settings where treatments are applied to groups. For example, a reading intervention may be implemented for all students in some schools and withheld from students in other schools. When such treatments are non-randomly allocated, outcomes across the treated and control groups may differ due to the treatment or due to baseline differences between groups. When this is the case, researchers can use statistical adjustment to make treated and control groups similar in terms of observed characteristics. Recent work in statistics has developed matching methods designed for contexts where treatments are clustered. This form of matching, known as multilevel matching, may be well suited to many education applications where treatments are assigned to schools. In this article, we provide an extensive evaluation of multilevel matching and compare it to multilevel regression modeling. We evaluate multilevel matching methods in two ways. First, we use these matching methods to recover treatment effect estimates from three clustered randomized trials using a within-study comparison design. Second, we conduct a simulation study. We find evidence that generally favors an analytic approach to statistical adjustment that combines multilevel matching with regression adjustment. We conclude with an empirical application.
2020
Lindsay C. Page, Matthew A. Lenard, and Luke Keele. 2020. “The Design of Clustered Observational Studies in Education.” AERA Open. Publisher's VersionAbstract
Clustered observational studies (COSs) are a critical analytic tool for educational effectiveness research. We present a design framework for the development and critique of COSs. The framework is built on the counterfactual model for causal inference and promotes the concept of designing COSs that emulate the targeted randomized trial that would have been conducted were it feasible. We emphasize the key role of understanding the assignment mechanism to study design. We review methods for statistical adjustment and highlight a recently developed form of matching designed specifically for COSs. We review how regression models can be profitably combined with matching and note best practice for estimates of statistical uncertainty. Finally, we review how sensitivity analyses can determine whether conclusions are sensitive to bias from potential unobserved confounders. We demonstrate concepts with an evaluation of a summer school reading intervention in Wake County, North Carolina.
J. Holbein, D.S. Hillygus, M. Lenard, C. Gibson-Davis, and D. Hill. 2020. “The Development of Students' Engagement in School, Community, and Democracy.” British Journal of Political Science, 50, 4, Pp. 1439-1457. Publisher's VersionAbstract
This article explores the origins of youth engagement in school, community and democracy. Specifically, it considers the role of psychosocial or non-cognitive abilities, like grit or perseverance. Using a novel original large-scale longitudinal survey of students linked to school administrative records and a variety of modeling techniques – including sibling, twin and individual fixed effects – the study finds that psychosocial abilities are a strong predictor of youth civic engagement. Gritty students miss less class time and are more engaged in their schools, are more politically efficacious, are more likely to intend to vote when they become eligible, and volunteer more. Our work highlights the value of psychosocial attributes in the political socialization of young people.
M. Lenard, M. Morrill, and J. Westall. 2020. “High School Start Times and Student Achievement: Looking Beyond Test Scores.” Economics of Education Review, 76. Publisher's VersionAbstract
The American Academy of Pediatrics recommends high schools begin after 8:30 AM to better align with the circadian rhythms of adolescents. Yet due to economic and logistic considerations such as transportation, athletics, and students’ after-school employment, the vast majority of high schools begin the school day considerably earlier. We leverage a quasi-natural experiment whereby five comprehensive high schools in a large and diverse school district moved start times forty minutes earlier to better coordinate with high schools already starting at 7:25 AM. In this setting, disruption effects from moving start times should exacerbate any harmful consequences of earlier start times. Early start times might negatively impact test scores, student engagement, and non-cognitive skill formation. We report on the effect of earlier start times on a broad range of outcomes, including mandatory ACT test scores, absenteeism, on-time progress in high school, and college-going. While we fail to find evidence of harmful effects on test scores, we do see a rise in absenteeism and tardiness rates, as well as higher rates of dropping out of high school. These results suggest that the harmful effects of early start times may not be well captured by considering test scores alone.
S.W. Hemelt and M.A. Lenard. 2020. “Math Acceleration in Elementary School: Access and Effects on Student Outcomes.” Economics of Education Review, 74. Publisher's VersionAbstract
This paper examines curricular acceleration in mathematics during elementary school using administrative data from a large, diverse school district that recently implemented a targeted, test-based acceleration policy. We first characterize access to advanced math and then estimate effects of acceleration in math on measures of short-run academic achievement as well as non- test-score measures of grit, engagement with schoolwork, future plans, and continued participation in the accelerated track. Experiences and effects of math acceleration differ markedly for girls and boys. Girls are less likely to be nominated for math acceleration and perform worse on the qualifying test, relative to boys with equivalent baseline performance. We find negative effects of acceleration on short-run retention of math knowledge for girls, but no such performance decay for boys. After initial exposure to accelerated math, girls are less likely than boys to appear in the accelerated track during late elementary school and at the start of middle school.
2019
S.W. Hemelt, M.A. Lenard, and C.G. Paeplow. 2019. “Building Bridges to Life after High School: Contemporary Career Academies and Student Outcomes.” Economics of Education Review, 68, Pp. 161-178. Publisher's VersionAbstract
Career academies serve an increasingly wide range of students. This paper examines the contemporary profile of students entering career academies in a large, diverse school district and estimates causal effects of participation in one of the district's well-regarded academies on a range of high school and college outcomes. Exploiting the lottery-based admissions process of this technology-focused academy, we find that academy enrollment increases the likelihood of high school graduation by about 8 percentage points and boosts rates of college enrollment for males but not females. Analysis of intermediate outcomes suggests that effects on attendance and industry-relevant certification at least partially mediate the overall high school graduation effect.
D. Carlson, E. Bell, M. Lenard, J. Cowen, and A. McEachin. 2019. “Socioeconomic-based School Assignment Policy and Racial Segregation Levels: Evidence from the Wake County Public School System.” American Educational Research Journal, 57, 1, Pp. 258-304 . Publisher's VersionAbstract
In the wake of legal challenges facing race-based integration, districts have turned tosocioeconomic integration in an attempt to achieve greater racial balance. Empirically, the extentto which these initiatives generate such balance is an open question. In this paper, we leveragethe school assignment system that the Wake County Public School System (WCPSS) employedto provide evidence on this issue. Although our results show that WCPSS’ socioeconomic-basedassignment policy had negligible effects on average levels of segregation across the district, itsubstantially reduced segregation for students who would have attended highly segregatedschools under a residence-based assignment policy. The policy also exposed these students topeers with different racial/ethnic backgrounds, higher achievement levels, and more advantagedneighborhood contexts.
2018
M.A. Lenard and P.A. Peña. 2018. “Maturity and minorities: The impact of redshirting on achievement gaps.” Education Economics. Publisher's VersionAbstract
There are sizable and pervasive academic achievement gaps between minority and non-minority students in the United States. Non-minority students – particularly boys – are more likely to enroll in school one year after they become eligible, a practice known as ‘redshirting.’ Consequently, non-minority students are on average more mature than minority students when they take standardized tests. Many studies have documented that differences in maturity at the moment of testing translate into large differences in test scores. Thus, differences in redshirting behavior across minority and non-minority students may be a contributing factor to achievement gaps. This study analyzes the effect of redshirting on achievement gaps using a reform in North Carolina that shifted the cutoff date for school eligibility in 2009 from October 16 to August 31. We use the reform to create an instrumental variable for redshirting behavior. Using data for eight cohorts of 3rd graders in the Wake County Public School System and a difference-in-differences approach, we estimate that redshirting increases the achievement gap by 28%–30% among boys born close to the cutoff date for school eligibility, and 3%–4% among all boys. For girls, the estimates are 8%–11% for those born close to the cutoff and 1% overall, but these estimates lack statistical significance. We discuss some policy implications of shifting the cutoff date for school eligibility – 14 states have done since 2000 – and growing redshirting rates.
S.D. Pimentel, L. Page, M. Lenard, and L. Keele. 2018. “Optimal Multilevel Matching Using Network Flows: An Application to a Summer Reading Intervention.” Annals of Applied Statistics, 12, 3, Pp. 1479-1505. Publisher's VersionAbstract
Many observational studies of causal effects occur in settings with clustered treatment assignment. In studies of this type, treatment is applied to entire clusters of units. For example, an educational in- tervention might be administered to all the students in a school. We develop a matching algorithm for multilevel data based on a network flow algorithm. Earlier work on multilevel matching relied on integer programming, which allows for balance targeting on specific covari- ates but can be slow with larger data sets. Although we cannot di- rectly specify minimal levels of balance for individual covariates, our algorithm is fast and scales easily to larger data sets. We apply this algorithm to assess a school-based intervention through which stu- dents in treated schools were exposed to a new reading program dur- ing summer school. In one variant of the algorithm, where we match both schools and students, we change the causal estimand through optimal subset matching to better maintain common support. In a second variant, we relax the common support assumption to preserve the causal estimand by only matching on schools. We find that the summer intervention does not appear to increase reading test scores. In a sensitivity analysis, however, we determine that an unobserved confounder could easily mask a larger treatment effect.