Please use this identifier to cite or link to this item:
http://hdl.handle.net/1942/34073
Title: | Certainty-Based Marking on Multiple-Choice Items: Psychometrics Meets Decision Theory | Authors: | Wu, Qian VAN ERUM, Monique AGTEN, Anouk Christiansen, Andres VANDENABEELE, Frank RIGO, Jean-Michel Janssen, Rianne |
Issue Date: | 2021 | Publisher: | SPRINGER | Source: | PSYCHOMETRIKA, 86(2), p. 518-543 | Abstract: | When a response to a multiple-choice item consists of selecting a single-best answer, it is not possible for examiners to differentiate between a response that is a product of knowledge and one that is largely a product of uncertainty. Certainty-based marking (CBM) is one testing format that requires examinees to express their degree of certainty on the response option they have selected, leading to an item score that depends both on the correctness of an answer and the certainty expressed. The expected score is maximized if examinees truthfully report their level of certainty. However, prospect theory states that people do not always make rational choices of the optimal outcome due to varying risk attitudes. By integrating a psychometric model and a decision-making perspective, the present study looks into the response behaviors of 334 first-year students of physiotherapy on six multiple-choice examinations with CBM in a case study. We used item response theory to model the objective probability of students giving a correct response to an item, and cumulative prospect theory to estimate their risk attitudes when students choose to report their certainty. The results showed that with the given CBM scoring matrix, students' choices of a certainty level were affected by their risk attitudes. Students were generally risk averse and loss averse when they had a high success probability on an item, leading to an under-reporting of their certainty. Meanwhile, they were risk seeking in case of small success probabilities on the items, resulting in the over-reporting of certainty. | Notes: | Janssen, R (corresponding author), Katholieke Univ Leuven, Ctr Educ Effectiveness & Evaluat, Dekenstr 2,Bus 3773, B-3000 Leuven, Belgium. rianne.janssen@kuleuven.be |
Other: | Janssen, R (corresponding author), Katholieke Univ Leuven, Ctr Educ Effectiveness & Evaluat, Dekenstr 2,Bus 3773, B-3000 Leuven, Belgium. rianne.janssen@kuleuven.be | Keywords: | certainty-based marking;certainty-based marking;multiple-choice questions;multiple-choice questions;IRT;IRT;cumulative prospect theory;cumulative prospect theory;hierarchical Bayesian estimation;hierar- chical Bayesian estimation | Document URI: | http://hdl.handle.net/1942/34073 | ISSN: | 0033-3123 | e-ISSN: | 1860-0980 | DOI: | 10.1007/s11336-021-09759-0 | ISI #: | WOS:000645520000001 | Rights: | © 2021 The Psychometric Society | Category: | A1 | Type: | Journal Contribution | Validations: | ecoom 2022 |
Appears in Collections: | Research publications |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
s11336-021-09759-0.pdf Restricted Access | Published version | 1.64 MB | Adobe PDF | View/Open Request a copy |
WEB OF SCIENCETM
Citations
4
checked on Oct 14, 2024
Page view(s)
62
checked on Sep 7, 2022
Download(s)
6
checked on Sep 7, 2022
Google ScholarTM
Check
Altmetric
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.