Measuring Instructional Sensitivity of Test Items: The Challenge of Incorporating Instructional Measures. Symposium: Conceptual and methodological challenges for valid inferences on Educational effectiveness

Instructional sensitivity relates to the extent to which a test or a single item reflects effects of classroom instruction (Polikoff, 2010). Substantiating instructional sensitivity is important when students’ test scores are used for evaluating teaching effectiveness: If instruments are not instructionally sensitive, valid inferences on teaching effectiveness based on test scores are precluded (Popham et al., 2014). While researchers have asserted that widely-used statistical sensitivity measures do not per se allow for valid inferences on instruction (e.g., van der Linden, 1981), it is unclear how to incorporate instructional measures into the evaluation of instructional sensitivity. Thus, we aim at investigating the extent to which instructional sensitivity measures are related to measures of (a) teaching quality and (b) content of instruction. We expect instructionally sensitive items to become easier with higher teaching quality and implementation of content in class. The sample for our analyses comprises 824 fifth-graders in 48 classrooms participating in a math assessment at two measurement occasions throughout the school year 2016/2017. Teaching quality was assessed via student questionnaires. In addition, teachers provided information on the implementation of teaching objectives. All analyses are carried out using the longitudinal multilevel IRT model to measure instructional sensitivity (Naumann, Hartig, & Hochweber, 2017). The results indicate (a) that the instructional measures partially explain statistical item sensitivity, (b) that it makes a difference which kind of measure is used to evaluate instructional sensitivity, and (c) that the relationship of instructional measures and items’ sensitivity depends on the item under investigation.