TY - JOUR
T1 - Direct behavior rating (DBR)
T2 - Generalizability and dependability across raters and observations
AU - Christ, Theodore J.
AU - Riley-Tillman, T. Chris
AU - Chafouleas, Sandra M.
AU - Boice, Christina H.
PY - 2010
Y1 - 2010
N2 - Generalizability theory was used to examine the generalizability and dependability of outcomes from two single-item Direct Behavior Rating (DBR) scales: DBR of actively manipulating and DBR of visually distracted. DBR is a behavioral assessment tool with specific instrumentation and procedures that can be used by a variety of service delivery providers (e.g., teacher, teacher aide, parent, etc.) to collect time-series data on student behavior. The purpose of this study was to extend the findings presented by Chafouleas et al. with an examination of DBR outcomes as they are generalized across raters and rating occasions. One hundred twenty-five undergraduates viewed and rated student behavior on video clips while the children engaged in an unsolvable Lego puzzle task. A series of decision studies were used to evaluate the effects of alternate assessment conditions (variable numbers of raters and rating occasions) and interpretive assumptions (definitions of the universe of generalization). Results support the general conclusion that ratings from individual or small groups of simultaneous raters, when generalized only to that specific individual or group of individuals, can approach reliability criteria for low- and high-stakes decisions. Implications are discussed.
AB - Generalizability theory was used to examine the generalizability and dependability of outcomes from two single-item Direct Behavior Rating (DBR) scales: DBR of actively manipulating and DBR of visually distracted. DBR is a behavioral assessment tool with specific instrumentation and procedures that can be used by a variety of service delivery providers (e.g., teacher, teacher aide, parent, etc.) to collect time-series data on student behavior. The purpose of this study was to extend the findings presented by Chafouleas et al. with an examination of DBR outcomes as they are generalized across raters and rating occasions. One hundred twenty-five undergraduates viewed and rated student behavior on video clips while the children engaged in an unsolvable Lego puzzle task. A series of decision studies were used to evaluate the effects of alternate assessment conditions (variable numbers of raters and rating occasions) and interpretive assumptions (definitions of the universe of generalization). Results support the general conclusion that ratings from individual or small groups of simultaneous raters, when generalized only to that specific individual or group of individuals, can approach reliability criteria for low- and high-stakes decisions. Implications are discussed.
KW - classroom behavior
KW - direct behavior rating
KW - direct observation
KW - educational measurement
KW - generalizability theory
KW - rating scale
KW - social behavior
UR - http://www.scopus.com/inward/record.url?scp=77956963834&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=77956963834&partnerID=8YFLogxK
U2 - 10.1177/0013164410366695
DO - 10.1177/0013164410366695
M3 - Article
AN - SCOPUS:77956963834
SN - 0013-1644
VL - 70
SP - 825
EP - 843
JO - Educational and Psychological Measurement
JF - Educational and Psychological Measurement
IS - 5
ER -