Publications

Working Paper
Kraft MA, Papay JP. Can Professional Environments in Schools Promote Teacher Development? Explaining Heterogeneity in Returns to Teaching Experience. Working Paper.Abstract
Although wide variation in teacher effectiveness is well established, much less is known about differences in teacher improvement over time. We document that average returns to teaching experience mask large variation across individual teachers, and across groups of teachers working in different schools. We examine the role of school context in explaining these differences using a measure of the professional environment constructed from teachers’ responses to state-wide surveys. Our analyses show that teachers working in more supportive professional environments improve their effectiveness more over time than teachers working in less supportive contexts. On average, teachers working in schools at the 75th percentile of professional environment ratings improved 20% more than teachers in schools at the 25th percentile after five years.
Download copy of the pdf
Kraft MA, Papay JP, Charner-Laird M, Johnson SM, Ng M, Reinhorn S. Committed to Their Students but in Need of Support: How School Context Influences Teacher Turnover in High-Poverty, Urban Schools. Working Paper.Abstract
Attracting and retaining effective teachers in high-poverty, urban schools remains a critical challenge. Some scholars interpret high turnover rates at these schools as evidence that teachers prefer to work with wealthier, whiter groups of students. Others argue that teachers are leaving behind the poor working conditions that tend to prevail in these schools. We interviewed 95 teachers and administrators in six high-poverty, urban schools in order to understand teachers’ views about their work with students and how school context influences their experience. We found that most teachers chose their schools, and stayed, because of their students. However, when schools failed to provide instructional supports, an orderly environment and extra assistance for students, teachers expressed frustration and their intentions to leave.
Download copy of pdf
Papay JP, Kraft MA. Productivity returns to experience in the teacher labor market: Methodological challenges and new evidence on long-term career growth. Working Paper.Abstract
We present new evidence on the relationship between employee productivity and job tenure using data from the teacher labor market. Econometric challenges require identifying assumptions to model the within-teacher returns to experience with teacher fixed effects. We describe the bias introduced by violations of two common assumptions, and we propose a third approach with a different and empirically-testable assumption. Consistent with past research, we find that teachers experience rapid productivity growth early in their careers. However, we find suggestive evidence of returns to experience later in the career, indicating that teachers continue to build human capital beyond these first years.
Download copy of PDF
Forthcoming
Kraft MA, Dougherty SM. The Effect of Teacher-Family Communication on Student Engagement: Evidence from a Randomized Field Experiment. Journal of Research on Educational Effectiveness. Forthcoming.Abstract
In this study, we seek to evaluate the efficacy of teacher communication with parents and students as a means of increasing student engagement. We estimate the causal effect of teacher communication by conducting a randomized field experiment in which 6th and 9th grade students were assigned to receive a daily phone call home and a text/written message during a mandatory summer school program. We find that frequent teacher-family communication immediately increased student engagement as measured by homework completion rates, on-task behavior and class participation. On average, teacher-family communication increased the odds a student completed their homework by 42% and decreased instances in which teachers had to redirect students’ attention to the task at hand by 25%. Class participation rates among 6th grade students increased by 49%, while communication appeared to have a small negative effect on 9th grade students’ willingness to participate. Drawing upon surveys and interviews with participating teachers and students, we identify three primary mechanisms through which communication likely affected engagement: stronger teacher-student relationships, expanded parental involvement, and increased student motivation.
Download copy of pdf
Kraft MA. How to Make Additional Time Matter: Integrating Individualized Tutorials into an Extended Day. Education Finance and Policy. Forthcoming.Abstract
Support for extending the school day has gained substantial momentum despite limited causal evidence that it increases student achievement. Existing evidence is decidedly mixed, in part, because of the stark differences in how schools use additional time. In this paper, I focus on the effect of additional time in school when that time is used for individualized tutorials. In 2005, MATCH Charter Public High School integrated two hours of individualized tutorials throughout an extended school day. The unanticipated implementation of this initiative and the school’s lottery enrollment policy allow me to use two complementary quasi-experimental methods to estimate program effects. I find that providing students with two hours of daily tutorials that are integrated into the school day and taught by full-time, recent college graduates increased achievement on 10th grade English language arts exams by 0.15- 0.25 standard deviations per year. I find no average effect in mathematics beyond the large gains students were already achieving, although quantile regression estimates suggest that the tutorials raised the lowest end of the achievement distribution in mathematics.
Download copy of pdf
Herlihy C, Karger E, Pollard C, Hill HC, Kraft MA, Williams M, Howard S. State and local efforts to investigate the validity and reliability of scores from teacher evaluation systems. Teachers College Record. Forthcoming.Abstract
Context: In the past two years, states have implemented sweeping reforms to their teacher evaluation systems in response to Race to the Top legislation and, more recently, NCLB waivers. With these new systems, policy-makers hope to make teacher evaluation both more rigorous and more grounded in specific job performance domains such as teaching quality and contributions to student outcomes. Attaching high stakes to teacher scores has prompted an increased focus on the reliability and validity of these scores. Teachers unions have expressed strong concerns about the reliability and validity of using student achievement data to evaluate teachers and the potential for subjective ratings by classroom observers to be biased. The legislation enacted by many states also requires scores derived from teacher observations and the overall systems of teacher evaluation to be valid and reliable. Focus of the study: In this paper, we explore how state education officials and their district and local partners plan to implement and evaluate their teacher evaluation systems, focusing in particular on states’ efforts to investigate the reliability and validity of scores emerging from the observational component of these systems. Research design: Through a document analysis and interviews with state education officials, we explore several issues that arise in observational systems, including the overall generalizability of teacher scores, the training, certification, and reliability of observers, and specifications regarding the sampling and number of lessons observed per teacher. Findings: Respondents’ reports suggest that states are attending to the reliability and validity of scores, but inconsistently; in only a few states does there appear to be a coherent strategy regarding reliability and validity in place. Conclusions: There remain a variety of system design and implementation decisions that states can optimize to increase the reliability and validity of their teacher evaluation scores. While a state may engage in auditing scores, for instance, it may miss the gains to reliability and validity that would accrue from periodic rater retraining and recertification, a stiff program of rater monitoring, and the use of multiple raters per teacher. Most troublesome are decisions about which and how many lessons to sample, which are either mandated legislatively, result from practical concerns or negotiations between stakeholders, or, at best case, rest on broad research not directly related to the state context. This suggests that states should more actively investigate the number of lessons and lesson sampling designs required to yield high-quality scores.
Download copy of pdf
2012
Johnson SM, Kraft MA, Papay JP. How context matters in high-need schools: The effects of teachers’ working conditions on their professional satisfaction and their students’ achievement. Teachers College Record. 2012;114(10):1-39.Abstract
Background/Context: Educational policymakers have begun to recognize the challenges posed by teacher turnover. Schools and students pay a price when new teachers leave the profession after only two or three years, just when they have acquired valuable teaching experience. Persistent turnover also disrupts efforts to build a strong organizational culture and to sustain coordinated instructional programs throughout the school. Retaining effective teachers is a particular challenge for schools that serve high proportions of low-income and minority students. Although some interpret these turnover patterns as evidence of teachers’ discontent with their students, recent large-scale quantitative studies provide evidence that teachers choose to leave schools with poor work environments, and that these conditions are most common in schools that minority and low-income students typically attend (Boyd et al., 2011; Ladd, 2009 & 2011; Borman & Dowling, 2008; Loeb, Darling-Hammond & Luczak, 2005). Thus, mounting evidence suggests that the seeming relationship between student demographics and teacher turnover is driven, not by teachers’ responses to their students, but by the conditions in which they must teach and their students are obliged to learn. Purpose/Objective/Research Question/Focus of Study: We build on this body of work by further examining how working conditions predict both teachers’ job satisfaction and their career plans. We use a broad conception of the context of teachers’ work, paying attention not only to narrowly defined working conditions, but also to the interpersonal and organizational contexts in which teachers work. We also extend Ladd’s (2009) analysis describing the relationship between the work context and student achievement. Advancing our understanding of this relationship is particularly important, given the increasing emphasis legislators place on evidence of student achievement when evaluating education policy. Specifically, we ask three research questions: (i) Do the conditions of work in Massachusetts public schools affect teachers’ satisfaction with their jobs and their career plans? (ii) Are schools with better conditions of work more successful in raising student performance than schools with less supportive working conditions? (iii) If the conditions of work are important, what elements of the work environment matter the most? Research Design: In this paper, we combine a statewide survey of school working conditions (Mass TeLLS) with demographic and student achievement data from Massachusetts. We examine three primary outcomes: teacher satisfaction, teacher career intentions, and student achievement growth. From different items on the Mass TeLLS, we construct a set of nine key elements that reflect the broad-based conditions in which teachers work. We fit standard regression models that describe the relationship between each outcome and both overall conditions of work and each element separately, modeling this relationship according to the properties of our outcome variables. Findings/Results: We find that measures of the school environment explain away much of the apparent relationship between teacher satisfaction and student demographic characteristics. The conditions in which teachers work matter a great deal to them and, ultimately, to their students. Teachers are more satisfied and plan to stay longer in schools that have a positive work context, independent of the school’s student demographic characteristics. Furthermore, although a wide range of working conditions matter to teachers, the specific elements of the work environment that matter the most to teachers are not narrowly conceived working conditions such as clean and well-maintained facilities or access to modern instructional technology. Instead, it is the social conditions—the school’s culture, the principal’s leadership, and relationships among colleagues—that predominate in predicting teachers’ job satisfaction and career plans. More importantly, providing a supportive context in which teachers can work appears to contribute to improved student achievement. We find that favorable conditions of work predict higher rates of student academic growth, even when we compare schools serving demographically similar groups of students. Conclusions/Recommendations: In short, we find that the conditions of teachers’ work matter a great deal. These results align with a growing body of work examining the organizational characteristics of the schools in which teachers work (Boyd et al., 2011; Ladd, 2011). Together, these studies suggest strongly that the high turnover rates of teachers in schools with substantial populations of low-income and minority students are driven largely by teachers fleeing the dysfunctional and unsupportive work environments in the schools to which low-income and minority students are most likely to be assigned. If public education is to provide effective teachers for all students, then the schools those students attend must become places that support effective teaching and learning across all classrooms.
Download copy of PDF
Hill HC, Charalambous CY, Blazar D, McGinn D, Kraft MA, Beisiegel M, Humez A, Litke E, Lynch K. Validating arguments for observational instruments: Attending to multiple sources of variation. Educational Assessment. 2012;17:1-19.Abstract
Measurement scholars have recently constructed validity arguments in support of a variety of educational assessments, including classroom observation instruments. In this article, we note that users must examine the robustness of validity arguments to variation in the implementation of these instruments. We illustrate how such an analysis might be used to assess a validity argument constructed for the Mathematical Quality of Instruction instrument, focusing in particular on the 20 effects of varying the rater pool, subject matter content, observation procedure, and district context. Variation in the subject matter content of lessons did not affect rater agreement with master scores, but the evaluation of other portions of the validity argument varied according to the composition of the rater pool, observation procedure, and district context. These results demonstrate the need for conducting such analyses, especially for classroom observation instruments that are subject to 25 multiple sources of variation.
Download copy of pdf
Hill HC, Charalambous CY, Kraft MA. When rater reliability is not enough: Teacher observation systems and a case for the G-study. Educational Researcher. 2012;41(2):56-64.Abstract
In recent years, interest has grown in using classroom observation as a means to several ends, including teacher development, teacher evaluation, and impact evaluations of classroom-based interventions. While educational practitioners and researchers have developed numerous observational instruments for these purposes, many fail to specify important criteria regarding their use. In this paper, we argue that for classroom observation to succeed in its aims, improved observational systems must be developed. These systems should include not only observational instruments, but also scoring designs capable of producing reliable and cost-efficient scores and processes for rater recruitment, training and certification. To illustrate how such a system might be developed and improved, we provide an empirical example that applies Generalizability Theory to data from a mathematics observational instrument.
Download copy of pdf
2007
Kraft MA. Toward a school-wide model of teaching for social justice: An examination of the best practices of two small public schools. Equity & Excellence in Education. 2007;40(1):77-86.Abstract
The purpose of this ethnographic study is to examine how a commitment to socially just pedagogies influences the core practices and policies of a school. This article presents a comprehensive description and analysis of two public middle schools committed to teaching for social justice: Urban Promise Academy and San Francisco Community School. From this exploratory research, I construct a school-wide model of teaching for social justice consisting of three central components: the integration of issues of social justice across the curriculum, the use of socially just teaching practices, and the creation of a socially just school community. This research further discusses the requirements and challenges of teaching for social justice in U.S. public schools. I argue for progressive educators to adopt practical, practitioner oriented models of schooling, such as the one presented in this article in order to more effectively advocate for school reform.
Download copy of PDF
2004
Kraft MA. The Guatemalan elections: Reshuffling the same business elites. Z Magazine [Internet]. 2004;17(2). Website