Working Paper
Guryan, J., Kim, J. S., Capotosto, L., Quinn, D. M., Kingston, H. C., Foster, L., & Cooc, N. (Working Paper). Can a Scaffolded Summer Reading Intervention Reduce Socioeconomic Gaps in Children’s Reading Comprehension Ability and Home Book Access? Results from a Randomized Experiment. Northwestern University Institute for Policy Research Working PaperAbstract

The researchers conducted a randomized experiment involving 824 third-grade children in 14 elementary schools (K-5) to examine the effects of a scaffolded summer reading intervention that provided books matched to children’s reading level and interests and teacher scaffolding in the form of end-of-year comprehension instruction. Within each school, children and teachers were randomly assigned to (a) a control condition involving six math lessons, (b) a treatment condition with six reading comprehension lessons and an after school family literacy event, and (c) an enhanced treatment that also included follow-up phone calls to parents. During summer vacation, children in the treatment conditions received two lesson books and eight matched books. A treatment by socioeconomic status (SES) interaction effect on reading comprehension posttests revealed larger positive effects for children in high poverty schools than children in moderate-high poverty schools. In addition, among a random subsample of children (n = 121) who were part of a home visit study, there were positive treatment effects on the quantity and the diversity of books at home and trends suggested larger effects for children from low SES families. The results highlight the variability in treatment effects across different school and family contexts.

Guryan, J., Kim, J. S., & Quinn, D. M. (Working Paper). Does reading during the summer build readingskills? Evidence from a randomized experiment in 463 classrooms. NBER Working PaperAbstract

There are large gaps in reading skills by family income among school-aged children in the United States. Correlational evidence suggests that reading skills are strongly related to the amount of reading students do outside of school. Experimental evidence testing whether this relationship is causal is lacking. We report the results from a randomized evaluation of a summer reading program called Project READS, which induces students to read more during the summer by mailing ten books to them, one per week. Simple intent-to-treat estimates show that the program increased reading during the summer, and show significant effects on reading comprehension test scores in the fall for third grade girls but not for third grade boys or second graders of either gender. Analyses that take advantage of within-classroom random assignment and cross-classroom variation in treatment effects show evidence that reading more books generates increases in reading comprehension skills, particularly when students read carefully enough to be able to answer basic questions about the books they read, and particularly for girls.

Quinn, D. M., Lynch, K., & Kim, J. S. (Working Paper). Summer Math Interventions and Income Status: A Meta-Analysis.Abstract

Academic summer programs are a popular strategy for providing compensatory education for low income students.  While early meta-analytic evidence suggested that summer school was more effective for middle income students than low income students (Cooper et al., 2000), a recent meta-analysis on summer reading programs found the opposite (Kim & Quinn, 2013).  In this paper, we extend Kim & Quinn (2013) by meta-analyzing 13 studies of summer math programs conducted between 1998 and 2011.  We find, consistent with Kim & Quinn's (2013) reading finding, that summer math programs were more effective for low income students than higher income students.  

Cooc, N., & Quinn, D. M. (2022). A Seasonal Analysis of Disparities in Academic Skills for Early Elementary School Children with Disabilities. The Elementary School Journal , 122, 502-533. Publisher's VersionAbstract
Studies examining seasonal variation in academic skills for children have focused on learning loss or stagnation during the summer, particularly for students from low-income or minoritized racial and ethnic backgrounds. In this study, we expand the literature to focus on another student population that may be susceptible to summer learning loss: children with disabilities. Using nationally representative data from the Early Childhood Longitudinal Study, Kindergarten Class of 2010–2011 (ECLS-K:2011), we examined changes in the academic skills of children with disabilities—defined in terms of having an individualized education plan or receiving special education services—relative to peers without disabilities during the school year and summer months. Results indicate that inequality in learning rates between both groups tends to widen more during the early school years, particularly in kindergarten, than summer. One policy implication of the findings is schools may need to focus more on kindergarten transition for children with disabilities.
Quinn, D. M., & Ho, A. D. (2021). Ordinal Approaches to Decomposing Between-group Test Score Disparities. Journal of Educational and Behavioral Statistics. Publisher's VersionAbstract
The estimation of test score “gaps” and gap trends plays an important role in monitoring educational inequality. Researchers decompose gaps and gap changes into within- and between-school portions to generate evidence on the role schools play in shaping these inequalities. However, existing decomposition methods assume an equal-interval test scale and are a poor fit to coarsened data such as proficiency categories. This leaves many potential data sources ill-suited for decomposition applications. We develop two decomposition approaches that overcome these limitations: an extension of V, an ordinal gap statistic, and an extension of ordered probit models. Simulations show V decompositions have negligible bias with small within-school samples. Ordered probit decompositions have negligible bias with large within-school samples but more serious bias with small within-school samples. More broadly, our methods enable analysts to (1) decompose the difference between two groups on any ordinal outcome into portions within- and between some third categorical variable and (2) estimate scale-invariant between-group differences that adjust for a categorical covariate.
Copur-Gencturk, Y., Thacker, I., & Quinn, D. M. (2021). K-8 Teachers’ Overall and Gender-Specific Beliefs About Mathematical Aptitude. International Journal of Science and Mathematics Education . Publisher's VersionAbstract
Teachers' beliefs play a significant role in students' academic attainment and career choices. Despite comparable attainment levels between genders, persistent stereotypes and beliefs that certain disciplines require innate ability and that men and women have different ability levels impede students' academic career paths. In this study, we examined the prevalence of U.S. mathematics teachers' explicit general and gender-specific beliefs about mathematical ability and identified which teacher characteristics were associated with these beliefs. An analysis of data from 382 K-8 teachers in the USA indicated that overall, teachers disagreed with the idea that general and gender-specific mathematical ability is innate, and agreed with the idea that hard work and dedication are required for success in mathematics. However, our findings indicate that those who believed mathematics requires brilliance also tended to think girls lacked this ability. We also found that teachers who were teaching mathematics to 11-to 14-year old students seemed to believe that mathematics requires innate ability compared with teachers who were teaching mathematics to 5-to 10-year-old students. In addition, more experienced teachers and teachers who worked with special education students seemed to believe less in the role of hard work in success in mathematics, which could have serious consequences for shaping their students' beliefs about their academic self-concept and future career-related decisions.
Chin, M. J., Quinn, D. M., Dhaliwal, T. K., & Lovison, V. S. (2020). Bias in the Air: A Nationwide Exploration of Teachers' Implicit Racial Attitudes, Aggregate Bias, and Student Outcomes. Educational Researcher. Publisher's VersionAbstract
Theory suggests that teachers’ implicit racial attitudes affect their students, but large-scale evidence on U.S. teachers’ implicit biases and their correlates is lacking. Using nationwide data from Project Implicit, we found that teachers’ implicit White/Black biases (as measured by the implicit association test) vary by teacher gender and race. Teachers’ adjusted bias levels are lower in counties with larger shares of Black students. In the aggregate, counties in which teachers hold higher levels of implicit and explicit racial bias have larger adjusted White/Black test score inequalities and White/Black suspension disparities.
Quinn, D. M. (2020). Experimental Evidence on Teachers' Racial Bias in Student Evaluation: The Role of Grading Scales. Educational Evaluation and Policy Analysis , 42, 375-392. Publisher's VersionAbstract
A vast research literature documents racial bias in teachers’ evaluations of students. Theory suggests bias may be larger on grading scales with vague or overly general criteria versus scales with clearly specified criteria, raising the possibility that well-designed grading policies may mitigate bias. This study offers relevant evidence through a randomized Web-based experiment with 1,549 teachers. On a vague grade-level evaluation scale, teachers rated a student writing sample lower when it was randomly signaled to have a Black author, versus a White author. However, there was no evidence of racial bias when teachers used a rubric with more clearly defined evaluation criteria. Contrary to expectation, I found no evidence that the magnitude of grading bias depends on teachers’ implicit or explicit racial attitudes.
Quinn, D. M. (2020). Experimental effects of "achievement gap" news reporting on viewers' racial stereotypes, inequality explanations, and inequality prioritization. Educational Researcher , 49, 482-492. Publisher's VersionAbstract
The “achievement gap” has long dominated mainstream conversations about race and education. Some scholars warn that the discourse around racial gaps perpetuates stereotypes and promotes the adoption of deficit-based explanations that fail to appreciate the role of structural inequities. I investigate through three randomized experiments. Results indicate that a TV news story about racial achievement gaps (vs. a control or counterstereotypical video) led viewers to express more exaggerated stereotypes of Black Americans as lacking education (Study 1 effect size = .30 SD; Study 2 effect size = .38 SD) and may have increased viewers’ implicit stereotyping of Black students as less competent than White students (Study 1 effect size = .22 SD; Study 2 effect size = .12 SD, ns). The video did not affect viewers’ explicit competence-related racial stereotyping, the explanations they gave for achievement inequalities, or their prioritization of ending achievement inequalities. After 2 weeks, the effect on stereotype exaggeration faded. Future research should probe how we can most productively frame educational inequality by race.
Kane, T. J., Blazar, D., Gehlbach, H., Greenberg, M., Quinn, D. M., & Thal, D. (2020). Can video technology improve teacher evaluations? An experimental study. Education Finance and Policy , 15, 397-427. Publisher's VersionAbstract
Teacher evaluation reform has been among the most controversial education reforms in recent years. It also is one of the most costly, in terms of the time teachers and principals must spend on classroom observations. We conducted a randomized field trial in four sites to evaluate whether substituting teacher-collected videos for in-person observations could improve the value of teacher observations for teachers, administrators, or students. Relative to teachers in the control group who participated in standard in-person observations, teachers in the video-based treatment group reported that post-observation meetings were more “supportive” and that they were more able to identify a specific practice they changed afterward. Treatment principals were able to shift their observation work to non-instructional times. The program also substantially increased teacher retention. Nevertheless, the intervention did not improve students' academic achievement or self-reported classroom experiences, either in the year of the intervention or for the next cohort of students. Following from the literature on observation and feedback cycles in low-stakes settings, we hypothesize that to improve student outcomes schools may need to pair video feedback with more specific supports for desired changes in practice.
Downey, D. B., Quinn, D. M., & Alcaraz, M. (2019). The Distribution of School Quality: Do Schools Serving Mostly White and High-SES Students Produce the Most Learning? Sociology of Education , 92 (4), 386-403. Publisher's VersionAbstract
What is schools’ role in the stratification system? One view is that schools are an important mechanism for perpetuating inequality because children from advantaged backgrounds (white and high socioeconomic) enjoy better school learning environments than their disadvantaged peers. But it is difficult to know this with confidence because children’s development is a product of both school and nonschool factors, making it a challenge to isolate school’s role. A novel approach for isolating school effects is to estimate the difference in learning when school is in versus out, what is called impact. Scholars employing this strategy have come to a remarkable conclusion—that schools serving disadvantaged children produce as much learning as those serving advantaged children. The empirical basis for this position is modest, however, and so we address several shortcomings of the previous research by analyzing a nationally representative sample of about 3,500 children in 270 schools from the Early Childhood Longitudinal Study–Kindergarten Cohort of 2011. With more comprehensive data and better scales, we also find no difference in impact on reading scores across schools serving poor or black children versus those serving nonpoor or white children. These patterns challenge the view that differences in school quality play an important role shaping achievement gaps and prompt us to reconsider theoretical positions regarding schools and inequality.
Quinn, D. M., & Stewart, A. M. (2019). Examining the Racial Attitudes of White Prek-12 Educators. The Elementary School Journal , 120 (2), 272-299. Publisher's VersionAbstract
As the racial/ethnic makeup of the U.S. student population becomes increasingly diverse, the teaching force has remained primarily white. Given this mismatch, and the consequences that educators’ racial attitudes have for students’ academic and developmental outcomes, it is important to understand the racial attitudes of white educators and how those attitudes differ from those of racially minoritized Americans. Using nationally representative data from 1975–2016, we find that compared with racially minoritized respondents, white educators are less supportive of government equalizing efforts, hold more negative racial stereotypes of African Americans, and express more social distance from minoritized groups. Through future research, we must develop a greater understanding of how these attitudes manifest in the classroom to affect students and how problematic attitudes of educators can be improved.
Quinn, D. M., Desruisseaux, T., & Nkansah-Amankra, A. (2019). "Achievement Gap" Language Affects Teachers' Issue Prioritization. Educational Researcher , 48 (7), 484-487. Publisher's VersionAbstract
The term “achievement gap” is regularly used to describe between-group differences in educational outcomes. However, critics of the term argue that it implies the problem is merely one of student performance and may depress support for policies aimed at structural solutions. We hypothesized that the phrase “racial achievement gap” would elicit lower levels of issue prioritization than the phrase “racial inequality in educational outcomes” due to the latter’s connotations of social justice. In a randomized survey experiment with a national teacher sample (N = 1,549), our hypothesis was confirmed. However, language did not affect teachers’ explanations for existing academic outcome disparities.
Quinn, D. M., & Le, Q. T. (2018). Are We Trending to More or Less Between-group Achievement Inequality Over the School Year and Summer? Comparing Across ECLS-K Cohorts. AERA Open , 4 (4), 1-19. Publisher's VersionAbstract
Scholars have argued that schools are “equalizers” because inequalities in test scores by race/ethnicity and socioeconomic status (SES) grow faster over summer vacation than over the school year. In this study, we use nationally representative data from the Early Childhood Longitudinal Study–Kindergarten Classes of 1998–1999 and 2010–2011 to examine the extent to which such patterns have changed over time. Results suggest that more between-group equalizing by race/ethnicity and SES occurred over kindergarten in the recent cohort. However, this was often followed by more inequality widening over summer and more widening or less narrowing over first grade in the latter cohort. The net result was that in recent years, inequality tended to widen more (Black-White) or narrow less (SES and Hispanic-White) over the first 2 years of schooling.
Quinn, D. M., Kane, T. J., Greenberg, M., & Thal, D. (2018). Effects of a Video-Based Teacher Observation Program on the De-privatization of Instruction: Evidence from a Randomized Experiment. Educational Administration Quarterly. Publisher's VersionAbstract
US schools have traditionally been characterized by teacher privacy and independence, yet theory and empirical work suggest that peer observation and support – or “de-privatized instruction” – can help improve pedagogical practice. In this randomized controlled trial, we investigate whether the introduction of video technology into a school, through a video-based teacher evaluation system called Best Foot Forward (BFF), led to instructional de-privatization. We find that BFF caused administrators to broker more peer support among teachers, made teachers more likely to share lesson videos with colleagues, led teachers to have more of their lessons seen by other teachers, and redistributed which teachers were providing instructional support to colleagues (with relatively newer teachers taking on a larger role in providing peer support). Results suggest that video technology may be an effective tool for efforts to improve instruction by increasing peer observation and support.
Quinn, D. M., & Kim, J. S. (2018). Experimental Effects of Program Management Approach on Teachers' Professional Ties and Social Capital. Educational Evaluation and Policy Analysis. , 40 (2), 196-218. Publisher's VersionAbstract

Theory and empirical work suggest that teachers’ social capital influences school improvement efforts.  Social ties are prerequisite for social capital, yet little causal evidence exists on how malleable factors, such as instructional management approaches, affect teachers’ ties.  In this cluster-randomized trial, we apply a decision-making perspective to compare a literacy intervention managed under a “fidelity-focused” approach, in which teachers were expected to implement researcher-designed procedures faithfully, versus a “structured adaptive” approach, in which teachers collaboratively planned program adaptations.  In the short term, the adaptive approach increased teachers’ accessing of intervention-related social capital, but decreased their accessing of social capital unrelated to the intervention.  Short-term effects varied based on participants’ role in the intervention.  No group differences were found on social capital measures one year later, suggesting that the structured adaptive approach did not make teachers more likely to form ties that would be useful outside of the intervention. 

Quinn, D. M. (2017). Racial Attitudes of PreK-12 and Postsecondary Educators: Descriptive Evidence from Nationally-representative Data. Educational Researcher , 46, 397-411. Publisher's VersionAbstract
PreK-12 and postsecondary educators’ racial attitudes have important consequences for students’ learning and development, yet we know little about educators’ racial attitudes overall, how their attitudes might differ from those of non-educators, or how attitudes might be changing over time. I investigate these questions using the nationally-representative General Social Survey. Some educators hold worrisome racial attitudes; yet compared to non-educators, educators are less opposed to governmental equalizing efforts, give more politically liberal explanations for racial inequalities, express less negative racial stereotypes, and report less social distance and collective resentments towards minoritized groups. Many educator/non-educator differences were explained by demographics, particularly education level. Time trends mostly show progress in Americans’ racial attitudes, with generally similar trends for educators and non-educators.
Kim, J. S., Burkhauser, M. A., Quinn, D. M., Guryan, J., Kingston, H. C., & Aleman, K. (2017). Effectiveness of Structured Teacher Adaptations to an Evidence-based Summer Literacy Program. Reading Research Quarterly. Publisher's VersionAbstract

We conducted a cluster randomized trial to examine the effectiveness of structured teacher adaptations to the implementation of an evidence-based summer literacy program that provided students with (a) books matched to their reading level and interests and (b) teacher scaffolding for summer reading in the form of end-of-year comprehension lessons and materials sent to students’ homes in the summer months. In this study, 27 high-poverty elementary schools (75-100% eligibility for free- or-reduced price lunch) were matched by prior reading achievement and poverty level and randomly assigned to one of two implementation conditions: a core treatment condition that directly replicated implementation procedures used in previous experiments or a core treatment with structured teacher adaptations condition. In the adaptations condition, teachers were organized into grade-level teams around a practical improvement goal and given structured opportunities to use their knowledge, experience, and local data to extend or modify program components for their students and local contexts. Students in the adaptations condition performed .12 standard deviations higher on a reading comprehension posttest than students in the core treatment. An implementation analysis suggests that fidelity to core program components was high in both conditions, while teachers in the adaptations condition primarily made changes that extended or modified program procedures and activities in acceptable ways. Adaptations primarily served to increase the level of family engagement and student engagement with summer books. These results suggest that structured teacher adaptations may enhance rather than diminish the effectiveness of an evidence-based summer literacy program.

Quinn, D. M., & Kim, J. S. (2017). Scaffolding Fidelity and Adaptation in Educational Program Implementation: Experimental Evidence from a Literacy Intervention. American Educational Research Journal , 54, 1187-1220. Publisher's VersionAbstract

In a common approach for scaling up effective educational practice, schools adopt evidence-based programs to be implemented with fidelity. An alternative approach assumes that programs should be adapted to local contexts. In this randomized trial of a reading intervention, we study a scaffolded sequence of implementation, in which schools first develop proficiency by implementing the program with fidelity before implementing structured adaptations. We find evidence supporting the scaffolded sequence: a fidelity-focused approach promoted learning and instructional change more so for teachers inexperienced with the intervention, while a structured adaptive approach was more effective for teachers experienced with the intervention. Students benefited more from the structured adaptive approach, but only when their teacher had prior experience with the fidelity-focused version.

Quinn, D. M., & McIntyre, J. (2017). Do Learning Rates Differ by Race/ethnicity over Kindergarten? Reconciling Results across Gain Score, First-Difference, and Random Effects Models. Economics of Education Review , 59, 81-86. Publisher's VersionAbstract


The question of whether students’ school-year learning rates differ by race/ethnicity is important for monitoring educational inequality.  Researchers applying different modeling strategies to the same data (the ECLS-K:99) have reached contrasting conclusions on this question.  We outline the similarities and differences across three common approaches to estimating gains and heterogeneity in gains: 1) a gain score model (with intercept), 2) a first-difference (FD) model (in some cases equivalent to regression-through-the-origin [RTO] and student fixed effects models), and 3) a student random effects (RE) model.  We show via simulation that FD/RTO and RE models produce estimates of learning rates, and group differences in learning rates, with more favorable RMSD compared to the gain score model with intercept.  Using data from the ECLS-K:99, we demonstrate that these precision differences lead to contrasting inferences regarding learning rate heterogeneity, and likely explain the inconsistencies across previous studies.