Publications

2023
Gilbert JB, Kim JS, Miratrix LW. Modeling Item-Level Heterogeneous Treatment Effects With the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions. Journal of Educational and Behavioral Statistics. 2023. Publisher's VersionAbstract
Analyses that reveal how treatment effects vary allow researchers, practitioners, and policymakers to better understand the efficacy of educational interventions. In practice, however, standard statistical methods for addressing heterogeneous treatment effects (HTE) fail to address the HTE that may exist within outcome measures. In this study, we present a novel application of the explanatory item response model (EIRM) for assessing what we term “item-level” HTE (IL-HTE), in which a unique treatment effect is estimated for each item in an assessment. Results from data simulation reveal that when IL-HTE is present but ignored in the model, standard errors can be underestimated and false positive rates can increase. We then apply the EIRM to assess the impact of a literacy intervention focused on promoting transfer in reading comprehension on a digital assessment delivered online to approximately 8,000 third-grade students. We demonstrate that allowing for IL-HTE can reveal treatment effects at the item-level masked by a null average treatment effect, and the EIRM can thus provide fine-grained information for researchers and policymakers on the potentially heterogeneous causal effects of educational interventions.
Relyea JE, Rich P, Kim JS, Gilbert JB. The COVID-19 impact on reading achievement growth of Grade 3–5 students in a US urban school district: variation across student characteristics and instructional modalities. Reading and Writing. 2023;36 (2) :317-346. Publisher's VersionAbstract
The current study aimed to explore the COVID-19 impact on reading achievement growth by Grade 3–5 students in a large urban school district in the U.S. and whether the impact differed by students’ demographic characteristics and instructional modality. Specifically, using administrative data from the school district, we investigated to what extent students made gains in reading during the 2020–2021 school year relative to the pre-COVID-19 typical school year in 2018–2019. We further examined whether the effects of students’ instructional modality on reading growth varied by demographic characteristics. Overall, students had lower average reading achievement gains over the 9-month 2020–2021 school year than the 2018–2019 school year with a learning loss effect size of 0.54, 0.27, and 0.28 standard deviation unit for Grade 3, 4, and 5, respectively. Substantially reduced reading gains were observed from Grade 3 students, students from high-poverty backgrounds, English learners, and students with disabilities. Additionally, findings indicate that among students with similar demographic characteristics, higher-achieving students tended to choose the fully remote instruction option, while lower-achieving students appeared to opt for in-person instruction at the beginning of the 2020–2021 school year. However, students who received in-person instruction most likely demonstrated continuous growth in reading over the school year, whereas initially higher-achieving students who received remote instruction showed stagnation or decline, particularly in the spring 2021 semester. Our findings support the notion that in-person schooling during the pandemic may serve as an equalizer for lower-achieving students, particularly from historically marginalized or vulnerable student populations.
2022
Asher CA, Scherer E, Kim JS. Using a Factorial Design to Maximize the Effectiveness of a Parental Text Messaging Intervention. Journal of Research on Educational Effectiveness. 2022;15 (3) :532-557. Publisher's VersionAbstract
 Parental text messaging interventions are growing in popularity to encourage at-home reading, school-attendance, and other educational behaviors. These interventions, which often combine multiple components, frequently demonstrate varying amounts of effectiveness, and researchers often cannot determine how individual components work alone or in combination with one another. Using a 2x2x3 factorial experiment, we investigate the effects of individual and interacted components from three behavioral levers to support summer reading: providing updated, personalized information; emphasizing different reading views; and goal setting. We find that the personalized information condition scored on average 0.03 SD higher on fall reading assessments. Texting effects on test scores were enhanced by messages that emphasized reading being useful for both entertainment and building skills compared to skill building alone or entertainment alone. These results continue to build our understanding that while text message can be an effective tool for parent engagement, the specific content of the message can lead to meaningful differences in the magnitude of the effects.
Kim JS, Burkhauser MA, Relyea JE, Gilbert JB, Scherer E, Fitzgerald J, Mosher D, McIntyre J. A longitudinal randomized trial of a sustained content literacy intervention from first to second grade: Transfer effects on students’ reading comprehension. Journal of Educational Psychology. 2022;115 (1) :73-98. Publisher's VersionAbstract
We developed a sustained content literacy intervention that emphasized building domain and topic knowledge from Grade 1 to Grade 2 and evaluated transfer effects on students’ reading comprehension outcomes. The Model of Reading Engagement (MORE) intervention emphasizes thematic lessons that provide an intellectual framework for helping students connect new learning to a general schema (i.e., how scientists study past events). A total of 30 elementary schools (N = 2,952 students; N = 144 teachers) were randomly assigned to a treatment or control group. Over 12 months, the treatment group students participated in (a) spring Grade 1 thematic content literacy lessons in science and social studies followed by wide reading of thematically related informational texts during summer, and (b) fall to spring Grade 2 thematic content literacy lessons in science. After implementation of Grade 1 thematic content literacy lessons and summer support for reading, treatment group students experienced smaller summer losses on a domain-general measure of reading than control group students. Following the sustained implementation of thematic content literacy lessons in science through Grade 2, treatment group students also outperformed their control group peers on a science content reading comprehension outcome (ES = .18). Furthermore, we found transfer effects on science content reading comprehension that varied by passage-item type (near-, mid-, and far-transfer passages determined by the inclusion and number of directly taught words in passages). A sustained content literacy intervention that aligns content and instruction across grades can help students transfer knowledge to novel reading comprehension tasks. (PsycInfo Database Record (c) 2023 APA, all rights reserved)
Kim JS, Burkhauser MA. Teaching for transfer can help young children read for understanding. Phi Delta Kappan. 2022;103 (8) :20-24. Publisher's VersionAbstract
To make progress in improving students’ reading comprehension, we need to rethink the very nature of reading comprehension — it’s not a skill and it requires background knowledge. James Kim and Mary Burkhauser explain that, to help students apply what they learn from one reading task to another reading task, educators should consider developing thematic lessons that organize knowledge around a schema, help students connect new knowledge to a schema, and measure students’ ability to transfer knowledge to novel comprehension tasks. They encourage educators to consider how they might use their knowledge of the role of schemas in reading comprehension to improve children’s ability to read for understanding.
2021
Kim JS, Burkhauser MA, Mesite L, Asher CA, Relyea JE, Fitzgerald J, Elmore J. Improving Reading Comprehension, Science Domain Knowledge, and Reading Engagement through a First-Grade Content Literacy Intervention. Journal of Educational Psychology. 2021;113 (1) :3-26. Publisher's Version
2019
*Troyer M, Kim JS, *Hale E, *Wantchekon K, *Armstrong C. Relations Among Intrinsic and Extrinsic Motivation, Reading Amount, and Comprehension: A Conceptual Replication. . Reading and Writing, An Interdisciplinary Journal. 2019;32 :1197-1218. Publisher's Version
*Qin W, Kingston HC, Kim JS. “What Does Retelling ‘Tell” about Children’s Reading Proficiency?". First Language. 2019;39 (2) :177-199. Publisher's Version
*Wantchekon K, Kim JS. Exploring the Relationship Between Reading Engagement and Reading Comprehension by Achievement Level. Reading & Writing Quarterly. 2019. Publisher's Version
Jones SM, LaRusso M, Kim HY, Selman R, Uccelli P, Barnes SP, Donovan S, Snow C. Experimental Effects of Word Generation on Vocabulary, Academic Language, Perspective Taking,and Reading Comprehension in High-Poverty Schools. Journal of Research on Educational Effectiveness. 2019. Publisher's Version
Kim JS, *Asher CA, Burkhauser M, *Mesite L, Leyva D. Using a Sequential Multiple Assignment Randomized Trial (SMART) to Develop an Adaptive K-2 Literacy Intervention with Personalized Print Texts and App-Based Digital Activities. AERA Open. 2019;5 (3) :1-18.
Kim JS. Making Every Study Count: Learning from Replication Failure to Improve Intervention Research. Educational Researcher. 2019;48 (9) :599-607. Publisher's VersionAbstract
Why, when so many educational interventions demonstrate positive impact in tightly controlled efficacy trials, are null results common in follow-up effectiveness trials? Using case studies from literacy, this article suggests that replication failure can surface hidden moderators—contextual differences between an efficacy and an effectiveness trial—and generate new hypotheses and questions to guide future research. First, replication failure can reveal systemic barriers to program implementation. Second, it can highlight for whom and in what contexts a program theory of change works best. Third, it suggests that a fidelity first and adaptation second model of program implementation can enhance the effectiveness of evidence-based interventions and improve student outcomes. Ultimately, researchers can make every study count by learning from both replication success and failure to improve the rigor, relevance, and reproducibility of intervention research.
2018
*Quinn DM, Kim JS. Experimental Effects of Program Management Approach on Teachers’ Professional Ties and Social Capital. Educational Evaluation and Policy Analysis. 2018;40 (2) :196-218. Publisher's Version
2017
Quinn DM, Kim JS. Experimental effects of program management approach on teachers’ professional ties and social capital. Educational Evaluation and Policy Analysis. 2017;40 (2) :196-218. Publisher's VersionAbstract
Theory and empirical work suggest that teachers’ social capital influences school improvement efforts. Social ties are prerequisite for social capital, yet little causal evidence exists on how malleable factors, such as instructional management approaches, affect teachers’ ties. In this cluster-randomized trial, we apply a decision-making perspective to compare a literacy intervention managed under a “fidelity-focused” approach, in which teachers were expected to implement researcher-designed procedures faithfully, versus a “structured adaptive” approach, in which teachers collaboratively planned program adaptations. In the short term, the adaptive approach increased teachers’ accessing of intervention-related social capital, but decreased their accessing of social capital unrelated to the intervention. Short-term effects varied based on participants’ role in the intervention. No group differences were found on social capital measures one year later, suggesting that the structured adaptive approach did not make teachers more likely to form ties that would be useful outside of the intervention.
Quinn DM, Kim JS. Scaffolding fidelity and adaptation in educational program implementation: Experimental evidence from a literacy intervention. American Educational Research Journal . 2017;54 (6) :1187-1220. Publisher's VersionAbstract
In a common approach for scaling up effective educational practice, schools adopt evidence-based programs to be implemented with fidelity. An alternative approach assumes that programs should be adapted to local contexts. In this randomized trial of a reading intervention, we study a scaffolded sequence of implementation in which schools first develop proficiency by implementing the program with fidelity before implementing structured adaptations. We find evidence supporting the scaffolded sequence: A fidelity-focused approach promoted learning and instructional change more so for teachers inexperienced with the intervention, while a structured adaptive approach was more effective for teachers experienced with the intervention. Students benefited more from the structured adaptive approach but only when their teacher had prior experience with the fidelity-focused version.
Kim JS, Burkhauser MA, Quinn DM, Guryan J, Kingston HC, Aleman K. Effectiveness of structured teacher adaptations to an evidence-based summer literacy program. Reading Research Quarterly. 2017;52 (4) :443-467. Publisher's VersionAbstract

The authors conducted a cluster-randomized trial to examine the effectiveness of structured teacher adaptations to the implementation of an evidence-based summer literacy program that provided students with (a) books matched to their reading level and interests and (b) teacher scaffolding for summer reading in the form of end-of-year comprehension lessons and materials sent to students’ homes in the summer months. In this study, 27 high-poverty elementary schools (75–100% eligibility for free or reduced-price lunch) were matched by prior reading achievement and poverty level and randomly assigned to one of two implementation conditions: a core treatment condition that directly replicated implementation procedures used in previous experiments, or a core treatment with structured teacher adaptations condition. In the adaptations condition, teachers were organized into grade-level teams around a practical improvement goal and given structured opportunities to use their knowledge, experience, and local data to extend or modify program components for their students and local contexts. Students in the adaptations condition performed 0.12 standard deviation higher on a reading comprehension posttest than students in the core treatment. An implementation analysis suggests that fidelity to core program components was high in both conditions and that teachers in the adaptations condition primarily made changes that extended or modified program procedures and activities in acceptable ways. Adaptations primarily served to increase the level of family engagement and student engagement with summer books. These results suggest that structured teacher adaptations may enhance rather than diminish the effectiveness of an evidence-based summer literacy program.

rrq-final-kim-etal.pdf
Cooc N, Kim JS. Peer influence on children’s reading skills: A social network analysis of elementary school classrooms. Journal of Educational Psychology. 2017;109 (5) :727-740. Publisher's VersionAbstract

Research has found that peers influence the academic achievement of children. However, the mechanisms through which peers matter remain underexplored. The present study examined the relationship between peers’ reading skills and children’s own reading skills among 4,215 total second- and third-graders in 294 classrooms across 41 schools. One innovation of the study was the use of social network analysis to directly assess who children reported talking to or seeking help from and whether children who identified peers with stronger reading skills experienced higher reading skills. The results indicated that children on average identified peers with stronger reading skills and the positive association between peer reading skills and children’s own reading achievement was strongest for children with lower initial levels of reading skills. The study has implications for how teachers can leverage the advantages of peers via in-class activities.

jep-peer_influence_in_reading.pdf
2016
Kim JS, Hemphill L, Troyer M, Thomson JM, Jones SM, LaRusso M, Donovan S. Engaging struggling adolescent readers to improve reading skills. Reading Research Quarterly. 2016. Publisher's VersionAbstract

This study examined the efficacy of a supplemental, multicomponent adolescent reading intervention for middle school students who scored below proficient on a state literacy assessment. Using a within-school experimental design, the authors randomly assigned 483 students in grades 6–8 to a business-as-usual control condition or to the Strategic Adolescent Reading Intervention (STARI), a supplemental reading program involving instruction to support word-reading skills, fluency, vocabulary, and comprehension, and peer talk to promote reading engagement and comprehension. The authors assessed behavioral engagement by measuring how much of the STARI curricular activities students completed during an academic school year, and collected intervention teachers’ ratings of their students’ reading engagement. STARI students outperformed control students on measures of word recognition (Cohen’s d = 0.20), efficiency of basic reading comprehension (Cohen’s d = 0.21), and morphological awareness (Cohen’s d = 0.18). Reading engagement in its behavioral form, as measured by students’ participation and involvement in the STARI curriculum, mediated the treatment effects on each of these three posttest outcomes. Intervention teachers’ ratings of their students’ emotional and cognitive engagement explained unique variance on reading posttests. Findings from this study support the hypothesis that (a) behavioral engagement fosters struggling adolescents’ reading growth, and (b) teachers’ perceptions of their students’ emotional and cognitive engagement further contribute to reading competence.

rrq-stari-final.pdf
Lynch K, Kim JS. Effects of a summer mathematics intervention for low-income children. Educational Evaluation and Policy Analysis. 2016. Publisher's VersionAbstract

Prior research suggests that summer learning loss among low-income children contributes to income-based gaps in achievement and educational attainment. We present results from a randomized experiment of a summer mathematics program conducted in a large, high-poverty urban public school district. Children in the third to ninth grade (N = 263) were randomly assigned to an offer of an online summer mathematics program, the same program plus a free laptop computer, or the control group. Being randomly assigned to the program plus laptop condition caused children to experience significantly higher reported levels of summer home mathematics engagement relative to their peers in the control group. Treatment and control children performed similarly on distal measures of academic achievement. We discuss implications for future research.

lynch_kim_summer_math_rct.pdf
Guryan J, Kim JS, Park KS. Motivation and incentives in education: Evidence from a summer reading experiment. Economics of Education Review. 2016;55 :1-20. Publisher's VersionAbstract
Policymakers and economists have expressed support for the use of incentives in educational settings. In this paper, rather than asking whether incentives work, we focus on a different question: For whom and under what conditions do incentives work? This question is particularly important because incentives’ promise relies on the idea that they might take the place of some cognitive failing or set of preferences that otherwise would have led students to make choices with large long-term benefits. In this paper, we explore whether that is the case. In the context of a summer reading program called Project READS, we test whether responsiveness to incentives is positively or negatively related to the student's baseline level of motivation to read. As a part of the program, elementary school students are mailed books weekly during the summer. We implemented this book-mailing program as a randomized experiment with three treatment arms. Students in the first treatment arm were mailed books as a part of the standard Project READS program. Students in the second treatment arm were mailed books as a part of Project READS, and were also offered an incentive to read the books they were mailed. Students in the third experimental group served as a control and were given books after posttesting occurred in the fall. We find that, if anything, more motivated readers are more responsive to incentives to read, suggesting that to the extent that incentives are effective, they may not effectively target the students whose behavior they are intended to change

Pages