Why, when so many educational interventions demonstrate positive impact in tightly controlled efficacy trials, are null results common in follow-up effectiveness trials? Using case studies from literacy, this article suggests that replication failure can surface hidden moderators—contextual differences between an efficacy and an effectiveness trial—and generate new hypotheses and questions to guide future research. First, replication failure can reveal systemic barriers to program implementation. Second, it can highlight for whom and in what contexts a program theory of change works best. Third, it suggests that a fidelity first and adaptation second model of program implementation can enhance the effectiveness of evidence-based interventions and improve student outcomes. Ultimately, researchers can make every study count by learning from both replication success and failure to improve the rigor, relevance, and reproducibility of intervention research.
Theory and empirical work suggest that teachers’ social capital influences school improvement efforts. Social ties are prerequisite for social capital, yet little causal evidence exists on how malleable factors, such as instructional management approaches, affect teachers’ ties. In this cluster-randomized trial, we apply a decision-making perspective to compare a literacy intervention managed under a “fidelity-focused” approach, in which teachers were expected to implement researcher-designed procedures faithfully, versus a “structured adaptive” approach, in which teachers collaboratively planned program adaptations. In the short term, the adaptive approach increased teachers’ accessing of intervention-related social capital, but decreased their accessing of social capital unrelated to the intervention. Short-term effects varied based on participants’ role in the intervention. No group differences were found on social capital measures one year later, suggesting that the structured adaptive approach did not make teachers more likely to form ties that would be useful outside of the intervention.
In a common approach for scaling up effective educational practice, schools adopt evidence-based programs to be implemented with fidelity. An alternative approach assumes that programs should be adapted to local contexts. In this randomized trial of a reading intervention, we study a scaffolded sequence of implementation in which schools first develop proficiency by implementing the program with fidelity before implementing structured adaptations. We find evidence supporting the scaffolded sequence: A fidelity-focused approach promoted learning and instructional change more so for teachers inexperienced with the intervention, while a structured adaptive approach was more effective for teachers experienced with the intervention. Students benefited more from the structured adaptive approach but only when their teacher had prior experience with the fidelity-focused version.
The authors conducted a cluster-randomized trial to examine the effectiveness of structured teacher adaptations to the implementation of an evidence-based summer literacy program that provided students with (a) books matched to their reading level and interests and (b) teacher scaffolding for summer reading in the form of end-of-year comprehension lessons and materials sent to students’ homes in the summer months. In this study, 27 high-poverty elementary schools (75–100% eligibility for free or reduced-price lunch) were matched by prior reading achievement and poverty level and randomly assigned to one of two implementation conditions: a core treatment condition that directly replicated implementation procedures used in previous experiments, or a core treatment with structured teacher adaptations condition. In the adaptations condition, teachers were organized into grade-level teams around a practical improvement goal and given structured opportunities to use their knowledge, experience, and local data to extend or modify program components for their students and local contexts. Students in the adaptations condition performed 0.12 standard deviation higher on a reading comprehension posttest than students in the core treatment. An implementation analysis suggests that fidelity to core program components was high in both conditions and that teachers in the adaptations condition primarily made changes that extended or modified program procedures and activities in acceptable ways. Adaptations primarily served to increase the level of family engagement and student engagement with summer books. These results suggest that structured teacher adaptations may enhance rather than diminish the effectiveness of an evidence-based summer literacy program.
Research has found that peers influence the academic achievement of children. However, the mechanisms through which peers matter remain underexplored. The present study examined the relationship between peers’ reading skills and children’s own reading skills among 4,215 total second- and third-graders in 294 classrooms across 41 schools. One innovation of the study was the use of social network analysis to directly assess who children reported talking to or seeking help from and whether children who identified peers with stronger reading skills experienced higher reading skills. The results indicated that children on average identified peers with stronger reading skills and the positive association between peer reading skills and children’s own reading achievement was strongest for children with lower initial levels of reading skills. The study has implications for how teachers can leverage the advantages of peers via in-class activities.
This study examined the efficacy of a supplemental, multicomponent adolescent reading intervention for middle school students who scored below proficient on a state literacy assessment. Using a within-school experimental design, the authors randomly assigned 483 students in grades 6–8 to a business-as-usual control condition or to the Strategic Adolescent Reading Intervention (STARI), a supplemental reading program involving instruction to support word-reading skills, fluency, vocabulary, and comprehension, and peer talk to promote reading engagement and comprehension. The authors assessed behavioral engagement by measuring how much of the STARI curricular activities students completed during an academic school year, and collected intervention teachers’ ratings of their students’ reading engagement. STARI students outperformed control students on measures of word recognition (Cohen’s d = 0.20), efficiency of basic reading comprehension (Cohen’s d = 0.21), and morphological awareness (Cohen’s d = 0.18). Reading engagement in its behavioral form, as measured by students’ participation and involvement in the STARI curriculum, mediated the treatment effects on each of these three posttest outcomes. Intervention teachers’ ratings of their students’ emotional and cognitive engagement explained unique variance on reading posttests. Findings from this study support the hypothesis that (a) behavioral engagement fosters struggling adolescents’ reading growth, and (b) teachers’ perceptions of their students’ emotional and cognitive engagement further contribute to reading competence.
Prior research suggests that summer learning loss among low-income children contributes to income-based gaps in achievement and educational attainment. We present results from a randomized experiment of a summer mathematics program conducted in a large, high-poverty urban public school district. Children in the third to ninth grade (N = 263) were randomly assigned to an offer of an online summer mathematics program, the same program plus a free laptop computer, or the control group. Being randomly assigned to the program plus laptop condition caused children to experience significantly higher reported levels of summer home mathematics engagement relative to their peers in the control group. Treatment and control children performed similarly on distal measures of academic achievement. We discuss implications for future research.
Policymakers and economists have expressed support for the use of incentives in educational settings. In this paper, rather than asking whether incentives work, we focus on a different question: For whom and under what conditions do incentives work? This question is particularly important because incentives’ promise relies on the idea that they might take the place of some cognitive failing or set of preferences that otherwise would have led students to make choices with large long-term benefits. In this paper, we explore whether that is the case. In the context of a summer reading program called Project READS, we test whether responsiveness to incentives is positively or negatively related to the student's baseline level of motivation to read. As a part of the program, elementary school students are mailed books weekly during the summer. We implemented this book-mailing program as a randomized experiment with three treatment arms. Students in the first treatment arm were mailed books as a part of the standard Project READS program. Students in the second treatment arm were mailed books as a part of Project READS, and were also offered an incentive to read the books they were mailed. Students in the third experimental group served as a control and were given books after posttesting occurred in the fall. We find that, if anything, more motivated readers are more responsive to incentives to read, suggesting that to the extent that incentives are effective, they may not effectively target the students whose behavior they are intended to change
To improve the reading comprehension outcomes of children in high-poverty schools, policymakers need to identify reading interventions that show promise of effectiveness at scale. This study evaluated the effectiveness of a low-cost and large-scale summer reading intervention that provided comprehension lessons at the end of the school year and stimulated home-based summer reading routines with narrative and informational books. We conducted a randomized controlled trial involving 59 elementary schools, 463 classrooms, and 6,383 second and third graders and examined outcomes on the North Carolina End-of-Grade (EOG) reading comprehension test administered nine months after the intervention, in the children's third- or fourth-grade year. We found that on this delayed outcome, the treatment had a statistically significant impact on children's reading comprehension, improving performance by .04 SD(standard deviation) overall and .05 SD in high-poverty schools. We also found, in estimates from an instrumental variables analysis, that children's participation in home-based summer book reading routines improved reading comprehension. The cost-effectiveness ratio for the intervention compared favorably to existing compensatory education programs that target high-poverty schools.
This study examines the effects of four types of reading comprehension questions – immediate, non-immediate, summary, and unanswerable questions – that linguistically diverse and predominantly low-income parents asked their fourth graders on children’s text retellings. One-hundred-twenty (N = 120) parent and child dyads participated in a home visit study in which they talked about narrative and informational texts. Moderation analyses indicated that immediate questions and non-immediate questions had a more positive effect on student retellings of an informational text and a narrative text, respectively, for less proficient than more proficient readers. These findings suggest that parents may be able to help their children, particularly less proficient readers, with text memory and text comprehension by asking specific types of questions.
A randomized trial involving 19 elementary schools (K–5) was conducted to replicate and extend two previous experimental studies of the effects of a voluntary summer reading program that provided (a) books matched to students’ reading levels and interests and (b) teacher scaffolding in the form of end-of-year comprehension lessons. Matched schools were randomly assigned to implement one of two lesson types. Within schools, students were randomly assigned to a control condition or one of two treatment conditions: a basic treatment condition replicating procedures used in the previous studies or an enhanced treatment condition that added teacher calls in the summer. During summer vacation, students in the treatment conditions received two lesson books and eight books matched to their reading level and interests. Overall, there were no significant treatment effects, and treatment effects did not differ across lesson type. However, there was a significant interaction between the treatment conditions and poverty measured at the school level. The effects of the treatments were positive for high-poverty schools (Cohen ’ s d = .08 and .11, respectively), defined as schools where 75–100% of the students were receiving free or reduced-price lunch ( FRL ). For moderate poverty schools (45–74% FRL ), the effects of the treatments were negative (Cohen ’ s d = −.11 and −.12, respectively). The results underscore the importance of looking at patterns of treatment effects across different contexts, settings, and populations.
This meta-analysis reviewed research on summer reading interventions conducted in the United States and Canada from 1998 to 2011. The synthesis included 41 classroom- and home- based summer reading interventions, involving children from kindergarten to Grade 8. Compared to control group children, children who participated in classroom interventions, involving teacher-directed literacy lessons, or home interventions, involving child-initiated book reading activities, enjoyed significant improvement on multiple reading outcomes. The magnitude of the treatment effect was positive for summer reading interventions that employed research-based reading instruction and included a majority of low-income children. Sensitivity analyses based on within-study comparisons indicated that summer reading interventions had significantly larger benefits for children from low-income backgrounds than for children from a mix of income backgrounds. The findings highlight the potentially positive impact of classroom- and home-based summer reading interventions on the reading comprehension ability of low- income children.
In this study, 72 secondary English teachers from the Santa Ana Unified School District were randomly assigned to participate in the Pathway Project, a cognitive strategies approach to teaching interpretive reading and analytical writing, or to a control condition involving typical district training focusing on teaching content from the textbook. Pathway teachers learned how to use an on-demand writing assessment to help mainstreamed English learners understand, interpret, and write analytical essays. In Year 2, treatment effects were replicated on an on-demand writing assessment (d = .67) and showed evidence of transfer to improved performance on a standardized writing test (d = .10). The results underscore the efficacy of a cognitive strategies reading/writing intervention for mainstreamed ELs in the secondary grades.