Research

2018
Kraft MA, Blazar D. Taking Teacher Coaching To Scale: Can Personalized Training Become Standard Practice?. Education Next. 2018;18 (4). Publisher's Version Download pdf here
Kraft MA, Blazar D, Hogan D. The Effect of Teacher Coaching on Instruction and Achievement: A Meta-Analysis of the Causal Evidence. Review of Educational Research. 2018;88 (4) :547-588. Publisher's VersionAbstract

Teacher coaching has emerged as a promising alternative to traditional models of professional development.  We review the empirical literature on teacher coaching and conduct meta-analyses to estimate the mean effect of coaching programs on teachers’ instructional practice and students’ academic achievement.  Combining results across 60 studies that employ causal research designs, we find pooled effect sizes of 0.49 standard deviations (SD) on instruction and 0.18 SD on achievement.  Much of this evidence comes from literacy coaching programs for pre-kindergarten and elementary school teachers.  Although these findings affirm the potential of coaching as a development tool, further analyses illustrate the challenges of taking coaching programs to scale while maintaining effectiveness.  Average effects from effectiveness trials of larger programs are only a fraction of the effects found in efficacy trials of smaller programs. We conclude by discussing ways to address scale-up implementation challenges and providing guidance for future causal studies.

Download pdf here
2017
Papay JP, Kraft MK. Developing Workplaces Where Teachers Stay, Improve, and Succeed. In: Teaching in Context: How Social Aspects of School and School Systems Shape Teachers’ Development & Effectiveness. Cambridge: Harvard Education Press ; 2017. pp. 15-35. Publisher's Version Download pdf here
Kraft MA. Engaging Parents Through Better Communication Systems. Educational Leadership. 2017;71 (September) :58-62. Publisher's VersionAbstract

New research shows that frequent, personalized outreach to parents works. How can schools support the practice?

Download pdf here
Kraft MA, Monti-Nussbaum M. Can schools enable parents to prevent summer learning loss? A text messaging field experiment to promote literacy skills. The ANNALS of the American Academy of Political and Social Science. 2017;674 (1) :85-112. Publisher's VersionAbstract

The vast differences in summer learning activities among children present a substantial challenge to providing equal educational opportunity in the United States. Most initiatives
aimed at reversing summer learning loss focus on school- or center-based programs. This study explores the potential of enabling parents to provide literacy development opportunities at home as a low-cost alternative. We conduct a randomized field trial of a summer text-messaging pilot program for parents focused on promoting literacy skills among first through fourth graders. We find positive effects on reading comprehension among third and fourth graders, with effect sizes of .21 to .29 standard deviations, but no effects for first and second graders. Texts also increased attendance at parent-teacher conferences but not at other school-related activities. Evidence to inform future efforts to reverse summer learning loss is
provided by parents’ responses to a follow-up survey.

Download pdf here
Steinberg M, Kraft MA. The Sensitivity of Teacher Performance Ratings to the Design of Teacher Evaluation Systems. Educational Researcher. 2017;46 (7) :378-396. Publisher's VersionAbstract

In recent years, states and districts have responded to federal incentives by instituting major reforms to their teacher evaluation systems. The passage of the Every Student Succeeds Act in 2015 now provides policymakers with even greater autonomy to redesign existing evaluation systems. Yet, little evidence exists to inform decisions about two key system design features – teacher performance measure weights and performance ratings thresholds. Using data from the Measures of Effective Teaching study, we conduct simulation-based analyses that illustrate the critical role that performance measure weights and ratings thresholds play in determining teachers’ summative evaluation ratings and the distribution of teacher proficiency rates. These findings offer insights to policymakers and administrators as they refine and possibly remake teacher evaluation systems. 

Download pdf here
Blazar D, Kraft MA. Teacher and teaching effects on students’ attitudes and behaviors. Educational Evaluation and Policy Analysis. 2017;39 (1) :146-170. Publisher's VersionAbstract

 

Research has focused predominantly on how teachers affect students’ achievement on tests despite evidence that a broad range of attitudes and behaviors are equally important to their long-term success. We find that upper-elementary teachers have large effects on self-reported measures of students’ self-efficacy in math, and happiness and behavior in class. Students’ attitudes and behaviors are predicted by teaching practices most proximal to these measures, including teachers’ emotional support and classroom organization. However, teachers who are effective at improving test scores often are not equally effective at improving students’ attitudes and behaviors. These findings lend empirical evidence to well-established theory on the multidimensional nature of teaching and the need to identify strategies for improving the full range of teachers’ skills.

Download pdf here
Charner-Laird M, Ng M, Johnson SM, Kraft MA, Papay JP, S.K R. Gauging goodness of fit: Teachers’ expectations for their instructional teams in high-poverty schools. American Journal of Education. 2017;123 (4) :553-584. Publisher's VersionAbstract

Teacher teams are increasingly common in urban schools. Here we analyze teachers'
responses to teams in six high-poverty schools. Teachers used two criteria to assess
teams' "goodness of fit" in meeting the demands of their work—whether their team
helped them teach better and whether it contributed to a better school. Their responses
differed notably by school, depending largely on the principal's approach to
implementation. In the three schools where teachers assessed teams favorably,
principals set a meaningful purpose for teachers' collaborative work, contributed
structural and professional expertise for their deliberations, and established a safe
environment for teachers' on-the-job growth.

Download pdf here
Kraft MA, Gilmour AF. Revisiting the Widget Effect: Teacher Evaluation Reforms and the Distribution of Teacher Effectiveness. Educational Researcher. 2017;46 (5) :234-249. Publisher's VersionAbstract

In 2009, The New Teacher Project (TNTP)’s The Widget Effect documented the failure to recognize and act on differences in teacher effectiveness. We revisit these findings by compiling teacher performance ratings across 24 states that adopted major reforms to their teacher evaluation systems. In the vast majority of these states, the percentage of teachers rated Unsatisfactory remains less than 1%. However, the full distributions of ratings vary widely across states with 0.7% to 28.7% rated below Proficient and 6% to 62% rated above Proficient. We present original survey data from an urban district illustrating that evaluators perceive more than three times as many teachers in their schools to be below Proficient than they rate as such. Interviews with principals reveal several potential explanations for these patterns.

Download pdf here
Kraft MA, Blazar DL. Individualized Coaching to Improve Teacher Practice Across Grades and Subjects: New Experimental Evidence. Educational Policy. 2017;31 (7) :1033-1068. Publisher's VersionAbstract

This paper analyzes a coaching model focused on classroom management skills and instructional practices across grade levels and subject areas. We describe the design and implementation of MATCH Teacher Coaching among an initial cohort of fifty-nine teachers working in New Orleans charter schools.  We evaluate the effect of the program on teachers’ instructional practices using a block randomized trial and find that coached teachers scored 0.59 standard deviations higher on an index of effective teaching practices comprised of observation scores, principal evaluations, and student surveys. We discuss implementation challenges and make recommendations for researcher-practitioner partnerships to address key remaining questions.

Download pdf here
2016
Papay JP, Kraft MA. The Myth of the Teacher Performance Plateau. Educational Leadership. 2016;73 (May) :36-42. Publisher's VersionAbstract

It’s almost accepted as fact that teachers don’t improve after their first few years on the job. New research challenges this common assumption.

Download pdf here
West MR, Kraft MA, Finn AS, Martin RE, Duckworth AL, Gabrieli CFO, Gabrieli JDE. Promise and Paradox: Measuring Students’ Non-cognitive Skills and the Impact of Schooling. Educational Evaluational and Policy Analysis. 2016;38 (1) :148-170. Publisher's VersionAbstract

We used self-report surveys to gather information on a broad set of non-cognitive skills from 1,368 8th-graders. At the student level, scales measuring conscientiousness, self-control, grit, and growth mindset are positively correlated with attendance, behavior, and test-score gains between 4th- and 8th-grade. Conscientiousness, self-control, and grit are unrelated to test-score gains at the school level, however, and students attending over-subscribed charter schools score lower on these scales than do students attending district schools. Exploiting admissions lotteries, we find positive impacts of charter school attendance on achievement and attendance but negative impacts on these non-cognitive skills. We provide suggestive evidence that these paradoxical results are driven by reference bias, or the tendency for survey responses to be influenced by social context.

Download pdf here
Papay JP, Kraft MA. The productivity costs of inefficient hiring practices: Evidence from late teacher hiring. Journal of Policy Analysis and Management. 2016;35 (4) :791-817. Publisher's VersionAbstract

We use matched employee-employer records from the teacher labor market to explore the trade-offs between the timing of hiring and match quality. Hiring teachers after the school year starts reduces student achievement by 0.042SD in mathematics and 0.026SD in reading. This reflects, in part, a temporary disruption effect in the first year. In mathematics, but not in reading, late-hired teachers remain persistently less effective, evidence of negative selection in the teacher labor market. Late hiring concentrates in schools that disproportionately serve disadvantaged student populations, contributing to challenges in ensuring an equitable distribution of educational resources across students.

Download pdf here
Kraft MA, Gilmour A. Can Principals Promote Teacher Development as Evaluators? A Case Study of Principals’ Views and Experiences. Educational Administration Quarterly. 2016;52 (5) :711-753. Publisher's VersionAbstract

Purpose: New teacher evaluation systems have expanded the role of principals as instructional leaders, but little is known about principals’ ability to promote teacher development through the evaluation process. We conducted a case study of principals’ perspectives on evaluation and their experiences implementing observation and feedback cycles to better understand whether principals feel as though they are able to promote teacher development as evaluators.

 

Research Methods: We conducted interviews with a stratified random sample of 24 principals in an urban district that recently implemented major reforms to its teacher evaluation system. We analyzed these interviews by drafting thematic summaries, coding interview transcripts, creating data-analytic matrices, and writing analytic memos.

 

Findings: We found that the evaluation reforms provided a common framework and language that helped facilitate principals’ feedback conversations with teachers. However, we also found that tasking principals with primary responsibility for conducting evaluations resulted in a variety of unintended consequences which undercut the quality of evaluation feedback they provided. We analyze five broad solutions to these challenges: strategically targeting evaluations, reducing operational responsibilities, providing principal training, hiring instructional coaches, and developing peer evaluation systems.

 

Implications: The quality of feedback teachers receive through the evaluation process depends critically on the time and training evaluators have to provide individualized and actionable feedback. Districts that task principals with primary responsibility for conducting observation and feedback cycles must attend to the many implementation challenges associated with this approach in order for next-generation evaluation systems to successfully promote teacher development.

Download pdf here
Kraft MA, Marinell WM, Yee D. School organizational contexts, teacher turnover, and student achievement: Evidence from panel data. American Educational Research Journal. 2016;53 (5) :1411-1499. Publisher's VersionAbstract

We study the relationship between school organizational contexts, teacher turnover, and student achievement in New York City (NYC) middle schools. Using factor analysis, we construct measures of four distinct dimensions of school contexts captured on the annual NYC School Survey. We identify credible estimates by isolating variation in organizational contexts within schools over time. We find that improvements in school leadership, academic expectations, teacher relationships, and school safety are all independently associated with corresponding reductions in teacher turnover. Increases in school safety and academic expectations for students also correspond to increases in student achievement. These results are robust to a range of potential threats to validity suggesting that our findings are likely driven by an underlying causal relationship.

Download pdf here
2015
Blazar DL, Kraft MA. Exploring Mechanisms of Effective Teacher Coaching: A Tale of Two Cohorts From a Randomized Experiment. Educational Evaluational and Policy Analysis. 2015;37 (4) :542–566. Publisher's VersionAbstract

Although previous research has shown that teacher coaching can improve teaching practices and student achievement, little is known about specific features of effective coaching programs. We estimate the impact of MATCH Teacher Coaching (MTC) on a range of teacher practices using a blocked randomized trial and explore how changes in the coaching model across two cohorts are related to program effects. Findings indicate large positive effects on teachers’ practices in cohort 1 but no effects in cohort 2. After ruling out explanations related to the research design, a set of exploratory analyses suggest that differential treatment effects may be attributable to differences in coach effectiveness and the focus of coaching across cohorts.

Download pdf here
Kraft MA. Teacher layoffs, teacher quality and student achievement: Evidence from a discretionary layoff policy. Education Finance and Policy. 2015;10 (4) :467-507. Publisher's VersionAbstract

Most teacher layoffs during the Great Recession were implemented following inverse-seniority policies. In this paper, I examine the implementation of a discretionary layoff policy in Charlotte Mecklenburg Schools. Administrators did not uniformly lay off the most or least senior teachers but instead selected teachers who were previously retired, late-hired, unlicensed, low-performing, or nontenured. Using quasi-experimental variation within schools across grades, I then estimate the differential effects of teacher layoffs on student achievement based on teacher seniority and effectiveness. Mathematics achievement in grades that lost an effective teacher, as measured by principal evaluations or value-added scores, decreased 0.05 to 0.11 standard deviations more than in grades that lost an ineffective teacher. In contrast, teacher seniority has little predictive power on the effects of layoffs. Simulation analyses show that the district selected teachers who were, on average, less effective than those teachers identified under an inverse-seniority policy, and also reduced job losses.

Download pdf here
Kraft MA, Rogers T. The underutilized potential of teacher-to-parent communication: Evidence from a field experiment. Economics of Education Review. 2015;47 :49-63. Publisher's VersionAbstract

We study an intervention designed to increase the effectiveness of parental involvement in their children’s education.  Each week we sent brief individualized messages from teachers to the parents of high school students in a credit recovery program.  This light-touch communication increased the probability students earned credits by 6.5 percentage points – a 41% reduction in the proportion failing to earn credit. This improvement resulted primarily from preventing drop-outs, rather than from reducing failure or dismissal rates.  The intervention shaped the content of parent-child conversations with messages emphasizing what students could improve, versus what students were doing well, producing the largest effects.  Our results illustrate the underutilized potential of communication policies with clear but reasonable expectations for teachers and program designs that make communication efficient and effective.

Download pdf here
Kraft MA, Papay JP, Charner-Laird M, Johnson SM, Ng M, Reinhorn S. Educating Amidst Uncertainty: The Organizational Supports Teachers Need to Serve Students in High-poverty, Urban Schools. Educational Administration Quarterly. 2015;51 (5) :753-790. WebsiteAbstract

Purpose

We examine how uncertainty, both about students and the context in which they are taught, remains a persistent condition of teachers’ work in high-poverty, urban schools. We describe six schools’ organizational responses to these uncertainties, analyze how these responses reflect more open- versus closed-system approaches, and examine how this orientation affects teachers’ work.

 

Research Methods

We draw on interviews with a diverse set of 95 teachers and administrators across a purposive sample of six high-poverty, urban schools in one district.  We analyzed these interviews by drafting thematic summaries, coding interview transcripts, creating data-analytic matrices, and writing analytic memos.

 

Findings

We find that students introduced considerable uncertainty into teachers’ work.  Although most teachers we spoke with embraced the challenges of their work and the expanded responsibilities that it entailed, they recognized that their individual efforts were not sufficient to succeed.  Teachers consistently spoke about the need for organizational responses that addressed the environmental uncertainty of working with students from low-income families whose experience in school often has been unsuccessful. We describe four types of organizational responses — coordinated instructional supports, systems to promote order and discipline, socio-emotional supports for students, and efforts to engage parents — and illustrate how these responses affected teachers’ ability to manage the uncertainty introduced by their environment. 

 

Conclusions

Traditional public schools are open systems, and require systematic organizational responses to address the uncertainty introduced by their environments.  Uncoordinated individual efforts alone are not sufficient to meet the needs of students in high-poverty urban communities.

Download pdf here
Papay JP, Kraft MA. Productivity returns to experience in the teacher labor market: Methodological challenges and new evidence on long-term career improvement. Journal of Public Economics. 2015;130 :105-119. Publisher's VersionAbstract

We present new evidence on the relationship between teacher productivity and job experience. Econometric challenges require identifying assumptions to model the within-teacher returns to experience with teacher fixed effects. We describe the bias introduced by violations of different identifying assumptions, including a new approach that we propose. Consistent with past research, we find that teachers experience rapid productivity improvement early in their careers. However, we find suggestive evidence of returns to experience later in the career, indicating that teachers continue to build human capital beyond these first years.

Download pdf here

Pages