Validating Arguments for Observational Instruments: Attending to Multiple Sources of Variation

Citation:

H. Hill, C. Charalambous, D. Blazar, D. McGinn, M. Kraft, M. Biesiegel, A. Humez, E. Litke, and K. Lynch. 2012. “Validating Arguments for Observational Instruments: Attending to Multiple Sources of Variation.” Educational Assessment, 2-3, Pp. 88-106. Publisher's Version

Abstract:

Measurement scholars have recently constructed validity arguments in support of a variety ofeducational assessments, including classroom observation instruments. In this article, we note thatusers must examine the robustness of validity arguments to variation in the implementation ofthese instruments. We illustrate how such an analysis mightbe used to assess a validity argumentconstructed for the Mathematical Quality of Instruction instrument, focusing in particular on the20effects of varying the rater pool, subject matter content, observation procedure, and district context.Variation in the subject matter content of lessons did not affect rater agreement with master scores,but the evaluation of other portions of the validity argument varied according to the compositionof the rater pool, observation procedure, and district context. These results demonstrate the needfor conducting such analyses, especially for classroom observation instruments that are subject to25multiple sources of variation.