TY - JOUR
T1 - Detecting Aberrant Behavior and Item Preknowledge
T2 - A Comparison of Mixture Modeling Method and Residual Method
AU - Wang, Chun
AU - Xu, Gongjun
AU - Shang, Zhuoran
AU - Kuncel, Nathan R
N1 - Publisher Copyright:
© 2018, 2018 AERA.
PY - 2018/8/1
Y1 - 2018/8/1
N2 - The modern web-based technology greatly popularizes computer-administered testing, also known as online testing. When these online tests are administered continuously within a certain “testing window,” many items are likely to be exposed and compromised, posing a type of test security concern. In addition, if the testing time is limited, another recognized aberrant behavior is rapid guessing, which refers to quickly answering an item without processing its meaning. Both cheating behavior and rapid guessing result in extremely short response times. This article introduces a mixture hierarchical item response theory model, using both response accuracy and response time information, to help differentiate aberrant behavior from normal behavior. The model-based approach is compared to the Bayesian residual-based fit statistic in both simulation study and two real data examples. Results show that the mixture model approach consistently outperforms the residual method in terms of correct detection rate and false positive error rate, in particular when the proportion of aberrance is high. Moreover, the model-based approach is also able to correctly identify compromised items better than residual method.
AB - The modern web-based technology greatly popularizes computer-administered testing, also known as online testing. When these online tests are administered continuously within a certain “testing window,” many items are likely to be exposed and compromised, posing a type of test security concern. In addition, if the testing time is limited, another recognized aberrant behavior is rapid guessing, which refers to quickly answering an item without processing its meaning. Both cheating behavior and rapid guessing result in extremely short response times. This article introduces a mixture hierarchical item response theory model, using both response accuracy and response time information, to help differentiate aberrant behavior from normal behavior. The model-based approach is compared to the Bayesian residual-based fit statistic in both simulation study and two real data examples. Results show that the mixture model approach consistently outperforms the residual method in terms of correct detection rate and false positive error rate, in particular when the proportion of aberrance is high. Moreover, the model-based approach is also able to correctly identify compromised items better than residual method.
KW - aberrant behavior
KW - item preknowledge
KW - item response theory
KW - mixture model
KW - person-fit
KW - response time
UR - http://www.scopus.com/inward/record.url?scp=85045135275&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85045135275&partnerID=8YFLogxK
U2 - 10.3102/1076998618767123
DO - 10.3102/1076998618767123
M3 - Article
AN - SCOPUS:85045135275
SN - 1076-9986
VL - 43
SP - 469
EP - 501
JO - Journal of Educational and Behavioral Statistics
JF - Journal of Educational and Behavioral Statistics
IS - 4
ER -