Research

2015
Kraft MA. How to Make Additional Time Matter: Integrating Individualized Tutorials into an Extended Day. Education Finance and Policy. 2015;10 (1) :81-116. WebsiteAbstract

Support for extending the school day has gained substantial momentum despite limited causal evidence that it increases student achievement. Existing evidence is decidedly mixed, in part, because of the stark differences in how schools use additional time. In this paper, I focus on the effect of additional time in school when that time is used for individualized tutorials. In 2005, MATCH Charter Public High School integrated two hours of individualized tutorials throughout an extended school day. The unanticipated implementation of this initiative and the school’s lottery enrollment policy allow me to use two complementary quasi-experimental methods to estimate program effects. I find that providing students with two hours of daily tutorials that are integrated into the school day and taught by full-time, recent college graduates increased achievement on 10th grade English language arts exams by 0.15- 0.25 standard deviations per year. I find no average effect in mathematics beyond the large gains students were already achieving, although quantile regression estimates suggest that the tutorials raised the lowest end of the achievement distribution in mathematics.

Download copy of pdf
2014
Fin A, Kraft MA, West MR, Leonard JA, Bisch CE, Martin RE, Sheridan MA, Gabrieli CF, Gabrieli JD. Cognitive Skills, Student Achievement Tests, and Schools. Psychological Science. 2014;25 (3) :736-744. Publisher's VersionAbstract

Cognitive skills predict academic performance, so schools that improve academic performance might also improve cognitive skills. To investigate the impact schools have on both academic performance and cognitive skills, we related standardized achievement test scores to measures of cognitive skills in a large sample (N=1,367) of 8th-grade students attending traditional, exam, and charter public schools. Test scores and gains in test scores over time correlated with measures of cognitive skills. Despite wide variation in test scores across schools, differences in cognitive skills across schools were negligible after controlling for 4th-grade test scores. Random offers of enrollment to over-subscribed charter schools resulted in positive impacts of such school attendance on math achievement, but had no impact on cognitive skills. These findings suggest that schools that improve standardized achievement tests do so primarily through channels other than cognitive skills.

Johnson SM, Reinhorn SK, Charner-Laird M, Kraft MA, Ng M, Papay JP. Ready to Lead, But How? Teachers’ Experiences in High-poverty Urban Schools. Teachers College Record. 2014;116 (10) :1-50. Publisher's VersionAbstract

We examine how teachers in six high-poverty, urban schools of one school district respond to and participate in leadership beyond their classrooms. We found that teachers wanted to participate in developing and implementing their school’s plans for change. When teachers believed that their principal took an inclusive approach to leadership, looking to them for ideas about how to improve the school, they were energized, committed to the joint effort and readily remained “in the game.” However, when teachers thought that their principal took an instrumental approach to leadership, expecting them to comply with fixed plans or to passively endorse administrative decisions, they expressed frustration and tended to withdraw to their classroom, sometimes intending to leave the school.

Download pdf here
Kraft MA, Papay JP. Can Professional Environments in Schools Promote Teacher Development? Explaining Heterogeneity in Returns to Teaching Experience. Educational Effectiveness and Policy Analysis. 2014;36 (4) :476-500. Publisher's VersionAbstract

Although wide variation in teacher effectiveness is well established, much less is known about differences in teacher improvement over time. We document that average returns to teaching experience mask large variation across individual teachers, and across groups of teachers working in different schools. We examine the role of school context in explaining these differences using a measure of the professional environment constructed from teachers’ responses to state-wide surveys. Our analyses show that teachers working in more supportive professional environments improve their effectiveness more over time than teachers working in less supportive contexts. On average, teachers working in schools at the 75th percentile of professional environment ratings improved 38% more than teachers in schools at the 25th percentile after ten years.

Download pdf here
Herlihy C, Karger E, Pollard C, Hill HC, Kraft MA, Williams M, Howard S. State and local efforts to investigate the validity and reliability of scores from teacher evaluation systems. Teachers College Record. 2014;116 (1) :1-28. Publisher's VersionAbstract

Context: In the past two years, states have implemented sweeping reforms to their teacher evaluation systems in response to Race to the Top legislation and, more recently, NCLB waivers. With these new systems, policy-makers hope to make teacher evaluation both more rigorous and more grounded in specific job performance domains such as teaching quality and contributions to student outcomes. Attaching high stakes to teacher scores has prompted an increased focus on the reliability and validity of these scores. Teachers unions have expressed strong concerns about the reliability and validity of using student achievement data to evaluate teachers and the potential for subjective ratings by classroom observers to be biased. The legislation enacted by many states also requires scores derived from teacher observations and the overall systems of teacher evaluation to be valid and reliable.

Focus of the study: In this paper, we explore how state education officials and their district and local partners plan to implement and evaluate their teacher evaluation systems, focusing in particular on states’ efforts to investigate the reliability and validity of scores emerging from the observational component of these systems.

Research design: Through a document analysis and interviews with state education officials, we explore several issues that arise in observational systems, including the overall generalizability of teacher scores, the training, certification, and reliability of observers, and specifications regarding the sampling and number of lessons observed per teacher.

Findings: Respondents’ reports suggest that states are attending to the reliability and validity of scores, but inconsistently; in only a few states does there appear to be a coherent strategy regarding reliability and validity in place.

Conclusions: There remain a variety of system design and implementation decisions that states can optimize to increase the reliability and validity of their teacher evaluation scores. While a state may engage in auditing scores, for instance, it may miss the gains to reliability and validity that would accrue from periodic rater retraining and recertification, a stiff program of rater monitoring, and the use of multiple raters per teacher. Most troublesome are decisions about which and how many lessons to sample, which are either mandated legislatively, result from practical concerns or negotiations between stakeholders, or, at best case, rest on broad research not directly related to the state context. This suggests that states should more actively investigate the number of lessons and lesson sampling designs required to yield high-quality scores.

Download copy of pdf
2013
Kraft MA, Dougherty SM. The Effect of Teacher-Family Communication on Student Engagement: Evidence from a Randomized Field Experiment. Journal of Research on Educational Effectiveness. 2013;6 (3) :199-222. WebsiteAbstract

In this study, we seek to evaluate the efficacy of teacher communication with parents and students as a means of increasing student engagement. We estimate the causal effect of teacher communication by conducting a randomized field experiment in which 6th and 9th grade students were assigned to receive a daily phone call home and a text/written message during a mandatory summer school program. We find that frequent teacher-family communication immediately increased student engagement as measured by homework completion rates, on-task behavior and class participation. On average, teacher-family communication increased the odds a student completed their homework by 42% and decreased instances in which teachers had to redirect students’ attention to the task at hand by 25%. Class participation rates among 6th grade students increased by 49%, while communication appeared to have a small negative effect on 9th grade students’ willingness to participate. Drawing upon surveys and interviews with participating teachers and students, we identify three primary mechanisms through which communication likely affected engagement: stronger teacher-student relationships, expanded parental involvement, and increased student motivation.

Download pdf here
2012
Hill HC, Charalambous CY, Blazar D, McGinn D, Kraft MA, Beisiegel M, Humez A, Litke E, Lynch K. Validating arguments for observational instruments: Attending to multiple sources of variation. Educational Assessment. 2012;17 (2-3) :88-106. WebsiteAbstract

Measurement scholars have recently constructed validity arguments in support of a variety of educational assessments, including classroom observation instruments. In this article, we note that users must examine the robustness of validity arguments to variation in the implementation of
these instruments. We illustrate how such an analysis might be used to assess a validity argument constructed for the Mathematical Quality of Instruction instrument, focusing in particular on the 20 effects of varying the rater pool, subject matter content, observation procedure, and district context. Variation in the subject matter content of lessons did not affect rater agreement with master scores, but the evaluation of other portions of the validity argument varied according to the composition of the rater pool, observation procedure, and district context. These results demonstrate the need for conducting such analyses, especially for classroom observation instruments that are subject to 25 multiple sources of variation.

Download copy of pdf
Hill HC, Charalambous CY, Kraft MA. When rater reliability is not enough: Teacher observation systems and a case for the G-study. Educational Researcher. 2012;41 (2) :56-64. WebsiteAbstract

In recent years, interest has grown in using classroom observation as a means to several ends, including teacher development, teacher evaluation, and impact evaluations of classroom-based interventions. While educational practitioners and researchers have developed numerous observational instruments for these purposes, many fail to specify important criteria regarding their use. In this paper, we argue that for classroom observation to succeed in its aims, improved observational systems must be developed. These systems should include not only observational instruments, but also scoring designs capable of producing reliable and cost-efficient scores and processes for rater recruitment, training and certification. To illustrate how such a system might be developed and improved, we provide an empirical example that applies Generalizability Theory to data from a mathematics observational instrument.

Download copy of pdf
Johnson SM, Kraft MA, Papay JP. How context matters in high-need schools: The effects of teachers’ working conditions on their professional satisfaction and their students’ achievement. Teachers College Record. 2012;114 (10) :1-39. WebsiteAbstract

Background/Context:
Educational policymakers have begun to recognize the challenges posed by teacher turnover. Schools and students pay a price when new teachers leave the profession after only two or three years, just when they have acquired valuable teaching experience. Persistent turnover also disrupts efforts to build a strong organizational culture and to sustain coordinated instructional programs throughout the school. Retaining effective teachers is a particular challenge for schools that serve high proportions of low-income and minority students. Although some interpret these turnover patterns as evidence of teachers’ discontent with their students, recent large-scale quantitative studies provide evidence that teachers choose to leave schools with poor work environments, and that these conditions are most common in schools that minority and low-income students typically attend (Boyd et al., 2011; Ladd, 2009 & 2011; Borman & Dowling, 2008; Loeb, Darling-Hammond & Luczak, 2005). Thus, mounting evidence suggests that the seeming relationship between student demographics and teacher turnover is driven, not by teachers’ responses to their students, but by the conditions in which they must teach and their students are obliged to learn.

Purpose/Objective/Research Question/Focus of Study:
We build on this body of work by further examining how working conditions predict both teachers’ job satisfaction and their career plans. We use a broad conception of the context of teachers’ work, paying attention not only to narrowly defined working conditions, but also to the interpersonal and organizational contexts in which teachers work. We also extend Ladd’s (2009) analysis describing the relationship between the work context and student achievement. Advancing our understanding of this relationship is particularly important, given the increasing emphasis legislators place on evidence of student achievement when evaluating education policy. Specifically, we ask three research questions: (i) Do the conditions of work in Massachusetts public schools affect teachers’ satisfaction with their jobs and their career plans? (ii) Are schools with better conditions of work more successful in raising student performance than schools with less supportive working conditions? (iii) If the conditions of work are important, what elements of the work environment matter the most?

Research Design:
In this paper, we combine a statewide survey of school working conditions (Mass TeLLS) with demographic and student achievement data from Massachusetts. We examine three primary outcomes: teacher satisfaction, teacher career intentions, and student achievement growth. From different items on the Mass TeLLS, we construct a set of nine key elements that reflect the broad-based conditions in which teachers work. We fit standard regression models that describe the relationship between each outcome and both overall conditions of work and each element separately, modeling this relationship according to the properties of our outcome variables.

Findings/Results:
We find that measures of the school environment explain away much of the apparent relationship between teacher satisfaction and student demographic characteristics. The conditions in which teachers work matter a great deal to them and, ultimately, to their students. Teachers are more satisfied and plan to stay longer in schools that have a positive work context, independent of the school’s student demographic characteristics. Furthermore, although a wide range of working conditions matter to teachers, the specific elements of the work environment that matter the most to teachers are not narrowly conceived working conditions such as clean and well-maintained facilities or access to modern instructional technology. Instead, it is the social conditions—the school’s culture, the principal’s leadership, and relationships among colleagues—that predominate in predicting teachers’ job satisfaction and career plans. More importantly, providing a supportive context in which teachers can work appears to contribute to improved student achievement. We find that favorable conditions of work predict higher rates of student academic growth, even when we compare schools serving demographically similar groups of students.

Conclusions/Recommendations:
In short, we find that the conditions of teachers’ work matter a great deal. These results align with a growing body of work examining the organizational characteristics of the schools in which teachers work (Boyd et al., 2011; Ladd, 2011). Together, these studies suggest strongly that the high turnover rates of teachers in schools with substantial populations of low-income and minority students are driven largely by teachers fleeing the dysfunctional and unsupportive work environments in the schools to which low-income and minority students are most likely to be assigned. If public education is to provide effective teachers for all students, then the schools those students attend must become places that support effective teaching and learning across all classrooms.

Download pdf here
2007
Kraft MA. Toward a school-wide model of teaching for social justice: An examination of the best practices of two small public schools. Equity & Excellence in Education. 2007;40 (1) :77-86. Publisher's VersionAbstract

The purpose of this ethnographic study is to examine how a commitment to socially just pedagogies influences the core practices and policies of a school. This article presents a comprehensive description and analysis of two public middle schools committed to teaching for social justice: Urban Promise Academy and San Francisco Community School. From this exploratory research, I construct a school-wide model of teaching for social justice consisting of three central components: the integration of issues of social justice across the curriculum, the use of socially just teaching practices, and the creation of a socially just school community. This research further discusses the requirements and challenges of teaching for social justice in U.S. public schools. I argue for progressive educators to adopt practical, practitioner oriented models of schooling, such as the one presented in this article in order to more effectively advocate for school reform.

Download copy of PDF

Pages