Working Paper
Beth Shueler, Catherine Armstrong Asher, Katherine Larned, Sarah Mehrotra, and Cynthia Pollard. Working Paper. “Improving Low-Performing Schools: A Meta-Analysis of Impact Evaluation Studies. Annenberg Institute EdWorkingPaper 20-274”. Publisher's VersionAbstract
The public narrative surrounding efforts to improve low-performing K-12 schools in the U.S. has been notably gloomy. Observers argue that either nothing works or we don’t know what works. At the same time, the federal government is asking localities to implement evidence-based interventions. But what is known empirically about whether school improvement works, how long it takes, which policies are most effective, and which contexts respond best to intervention? We meta-analyze 141 estimates from 67 studies of turnaround policies implemented post-NCLB. On average, these policies have had a moderate positive effect on math but no effect on ELA achievement as measured by high-stakes exams. We find evidence of positive impacts on low-stakes exams in STEM and humanities subjects and no evidence of harm on non-test outcomes. Some elements of reform, namely extended learning time and teacher replacements, predict greater effects. Contexts serving majority-Latinx populations have seen the largest improvements.
Catherine Armstrong Asher, Ethan Scherer, and James Kim. Working Paper. “Using a Factorial Design to Maximize the Effectiveness of a Parental Text Messaging Intervention. Annenberg Institute EdWorkingPaper 20-305.”. Publisher's VersionAbstract
Parental text messaging interventions are growing in popularity to encourage at-home reading, school-attendance, and other educational behaviors. These interventions, which often combine multiple components, frequently demonstrate varying amounts of effectiveness, and researchers often cannot determine how individual components work alone or in combination with one another. Using a 2x2x3 factorial experiment, we investigate the effects of individual and interacted components from three behavioral levers to support summer reading: providing updated, personalized information; emphasizing different reading views; and goal setting. We find that the personalized information condition scored 0.03 SD higher on fall reading assessments. Test score effects were enhanced by messages that emphasized reading being useful for both entertainment and building skills compared to skill building alone or entertainment alone.
James S. Kim, Mary Burkhauser, Laura Mesite, Catherine A. Asher, Jackie Eunjung Relyea, Jill Fitzgerald, and Jeff Elmore. 2020. “Improving reading comprehension, science domain knowledge, and reading engagement through a first-grade content literacy intervention.” Journal of Educational Psychology. Publisher's VersionAbstract
This study investigated the effectiveness of the Model of Reading Engagement (MORE), a content literacy intervention, on first graders’ science domain knowledge, reading engagement, and reading comprehension. The MORE intervention emphasizes the role of domain knowledge and reading engagement in supporting reading comprehension. MORE lessons included a 10-day thematic unit that provided a framework for students to connect new learning to a meaningful schema (i.e., Arctic animal survival) and to pursue mastery goals for acquiring domain knowledge. A total of 38 first-grade classrooms (N = 674 students) within 10 elementary schools were randomly assigned to (a) MORE at school (MS), (b) MORE at home, (MS-H), in which the MS condition included at-home reading, or (c) typical instruction. Since there were minimal differences in procedures between the MS and MS-H conditions, the main analyses combined the two treatment groups. Findings from hierarchical linear models revealed that the MORE intervention had a positive and significant effect on science domain knowledge, as measured by vocabulary knowledge depth (ES = .30), listening comprehension (ES = .40), and argumentative writing (ES = .24). The MORE intervention effects on reading engagement as measured by situational interest, reading motivation, and task orientations were not statistically significant. However, the intervention had a significant, positive effect on a distal measure of reading comprehension (ES = .11), and there was no evidence of treatment-by-aptitude interaction effects. Content literacy can facilitate first graders’ acquisition of science domain knowledge and reading comprehension without contributing to Matthew effects. 
James S. Kim, Catherine A. Asher, Mary Burkhauser, Laura Mesite, and Diana Leyva. 2019. “Using a sequential multiple assignment randomized trial (SMART) to develop an adaptive K-2 literacy intervention with personalized print texts and app-based digital activities. .” AERA Open, 5, 3, Pp. 1-18. Publisher's VersionAbstract
This study employs a sequential multiple assignment randomized trial (SMART) design to develop an adaptive intervention with personalized print and digital content for kindergarten to Grade 2 children (n = 273). In Stage 1, we ask whether it is better for children to receive an adaptive intervention based on (a) 10 conceptually coherent texts or (b) 10 leveled texts on a range of topics. In Stage 2, we ask how best to encourage nonresponding children. Findings indicate that children who received either conceptually coherent texts or leveled texts performed similarly on reading comprehension posttests, while augmenting and intensifying follow-up with gamification of the app and text messages to parents improved comprehension outcomes for nonresponders. Descriptively, we find that only 26% (n = 71) of parents accessed the app, highlighting the need for better implementation procedures to increase take up of app-based digital activities.
Margaret Troyer, James S. Kim, Elizabeth Hale, Kristia A. Wantchekon, and Catherine Armstrong. 2018. “Relations among intrinsic and extrinsic reading motivation, reading amount, and comprehension: a conceptual replication.” Reading and Writing, 32, Pp. 1197-1218. Publisher's VersionAbstract
Children’s motivation to read is a strong predictor of their reading comprehension. However, some recent research has suggested that the relationship between reading motivation and reading comprehension may be mediated through the amount that students read. This study attempts a conceptual replication of several existing models that explore the relationship among children’s reading motivations, out-ofschool reading amount, and reading comprehension, using a large sample of over 4000 third- through ffth-graders in 59 U.S. elementary schools. Consistent with prior research, several control variables, including children’s prior reading comprehension ability, gender, and socioeconomic status, directly contributed to later reading comprehension. Results also replicated positive associations between intrinsic reading motivation, reading amount and reading comprehension, and negative associations between extrinsic reading motivation, reading amount and reading comprehension. Using structural equation models, our analyses found no evidence that the relationship between children’s intrinsic and extrinsic reading motivation and later reading comprehension was either partially or fully mediated by reading amount. This suggests that it is critical to attend to context-specifc determinants of motivation and reading amount, including students’ background characteristics and quality of texts read. Furthermore, this study underscores the importance of replicating methods used by original researchers to confrm and disconfrm hypotheses, and of conducting research with large and diverse samples that enhance the generalizability of results.
James S. Kim, Catherine M. Armstrong, and Thomas Kelley-Kemple. 2017. Practices matter: major findings from the Charlotte-Mecklenburg Schools (CMS) teacher literacy survey. Cambridge, MA: READS Lab, Harvard Graduate School of Education. PracticesMatter_fullreport.pdf
Robin T. Jacob, Catherine Armstrong, A. Brooks Bowden, and Yilin Pan. 2016. “Leveraging volunteers: an experimental evaluation of a tutoring program for struggling readers.” Journal of Research on Educational Effectiveness, 9, S1, Pp. 67-92. Publisher's VersionAbstract
This study evaluates the impacts and costs of the Reading Partners program, which uses community volunteers to provide one-on-one tutoring to struggling readers in under-resourced elementary schools. The evaluation uses an experimental design. Students were randomly assigned within 19 different Reading Partners sites to a program or control condition to answer questions about the impact of the program on student reading proficiency. A cost study, using a subsample of six of the 19 study sites, explores the resources needed to implement the Reading Partners program as described in the evaluation. Findings indicate that the Reading Partners program has a positive and statistically significant impact on all three measures of reading proficiency assessed with an effect size equal to around 0.10. The cost study findings illustrate the potential value of the Reading Partners program from the schools' perspective because the financial and other resources required by the schools to implement the program are low. Additionally, the study serves as an example of how evaluations can rigorously examine both the impacts and costs of a program to provide evidence regarding effectiveness.