Investigating standard-setter cognition
Resource type
Conference Paper not in Proceedings
Status
Published
Recommended form of citation (APA)
Moser-Frötscher, D., Hollenstein, S., Hilbe, R. & Nesci, N. (2024, October 18-19). Investigating standard-setter cognition [Confernence paper]. Responding to the CEFR Alignment Handbook: Sharing experience of alignment activities and reflecting on lessons learned. 18.-19.10.2024, Barcelona.
PHSG Organisation name
License Condition
All rights reserved
Proforis OA-status
metadata only (bibliographisch)
Topic PHSG
Sprachliche und literarische Bildung::Sprachliche Bildung
Pädagogische Psychologie::Bildungsevaluation
Pädagogische Psychologie::Lernfördersysteme
Fields of Science and Technology (OECD)
Humanities::Languages and Literature::General language studies
Social sciences::Educational sciences::Education, general (including training, pedagogy, didactics)
Abstract
Decisions made by standard-setters directly affect the thresholds to be attained by students. It is therefore crucial to better understand the black box of standard-setting (McGinty, 2010). While research into standard-setters’ decision-making has examined group discussions (Papageorgiou, 2010), standard setters’ thought processes have, to our knowledge, not been investigated at the individual level. Furthermore, decision research postulates that complex decisions are shaped by individuals’ decision-making style and their preferential mode of processing (Newell & Bröder, 2008).
We report on an alignment case study in progress. The research questions are as follows:
RQ1: What are standard-setters’ thought processes while deciding on the CEFR level of reading and listening tasks?</li>
RQ2: To what extent do standard-setters’ decision-making style and preferred processing mode influence rating quality and rating behaviour? </li>
Data will be collected in two virtual standard-setting workshops (reading and listening) for a standardized English L2 achievement test at lower secondary level, employing the descriptormatching method using ordered booklets (85 items) (Ferrara & Lewis, 2012). During the rating phase, concurrent think-aloud protocols will be recorded and subsequently coded using MaxQDA to investigate standard-setters’ thought processes. To address RQ2, standard-setters (n=30) will be administered the Rational-Experiential Inventory 40 (Pacini & Epstein, 1999) and the General Decision Making Style Inventory (Scott & Bruce, 1995), validated questionnaires gauging decisionmaking style and preferred processing mode, respectively. Correlations will be calculated between scores derived from the questionnaires and rating metrics established through ManyFacets Rasch Measurement analyses. In our presentation, we will discuss preliminary findings and implications.
We report on an alignment case study in progress. The research questions are as follows:
RQ1: What are standard-setters’ thought processes while deciding on the CEFR level of reading and listening tasks?</li>
RQ2: To what extent do standard-setters’ decision-making style and preferred processing mode influence rating quality and rating behaviour? </li>
Data will be collected in two virtual standard-setting workshops (reading and listening) for a standardized English L2 achievement test at lower secondary level, employing the descriptormatching method using ordered booklets (85 items) (Ferrara & Lewis, 2012). During the rating phase, concurrent think-aloud protocols will be recorded and subsequently coded using MaxQDA to investigate standard-setters’ thought processes. To address RQ2, standard-setters (n=30) will be administered the Rational-Experiential Inventory 40 (Pacini & Epstein, 1999) and the General Decision Making Style Inventory (Scott & Bruce, 1995), validated questionnaires gauging decisionmaking style and preferred processing mode, respectively. Correlations will be calculated between scores derived from the questionnaires and rating metrics established through ManyFacets Rasch Measurement analyses. In our presentation, we will discuss preliminary findings and implications.
| Name of the event | Conference Host | Place of the event | Start date of the event | End date of the event |
Responding to the CEFR Alignment Handbook: Sharing experience of alignment activities and reflecting on lessons learned | BlanquernaUniversitat Ramon Llull | Barcelona | October 18, 2024 | October 19, 2024 |
Access Rights
metadata only (bibliographisch)
License Condition
All rights reserved
Rights Holder
Author(s)