TY - JOUR
T1 - ManyClasses 1
T2 - Assessing the Generalizable Effect of Immediate Feedback Versus Delayed Feedback Across Many College Classes
AU - Fyfe, Emily R.
AU - de Leeuw, Joshua R.
AU - Carvalho, Paulo F.
AU - Goldstone, Robert L.
AU - Sherman, Janelle
AU - Admiraal, David
AU - Alford, Laura K.
AU - Bonner, Alison
AU - Brassil, Chad E.
AU - Brooks, Christopher A.
AU - Carbonetto, Tracey
AU - Chang, Sau Hou
AU - Cruz, Laura
AU - Czymoniewicz-Klippel, Melina
AU - Daniel, Frances
AU - Driessen, Michelle
AU - Habashy, Noel
AU - Hanson-Bradley, Carrie L.
AU - Hirt, Edward R.
AU - Carbonell, Virginia Hojas
AU - Jackson, Daniel K.
AU - Jones, Shay
AU - Keagy, Jennifer L.
AU - Keith, Brandi
AU - Malmquist, Sarah J.
AU - McQuarrie, Barry
AU - Metzger, Kelsey J.
AU - Min, Maung K.
AU - Patil, Sameer
AU - Patrick, Ryan S.
AU - Pelaprat, Etienne
AU - Petrunich-Rutherford, Maureen L.
AU - Porter, Meghan R.
AU - Prescott, Kristina
AU - Reck, Cathrine
AU - Renner, Terri
AU - Robbins, Eric
AU - Smith, Adam R.
AU - Stuczynski, Phil
AU - Thompson, Jaye
AU - Tsotakos, Nikolaos
AU - Turk, Judith K.
AU - Unruh, Kyle
AU - Webb, Jennifer D.
AU - Whitehead, Stephanie N.
AU - Wisniewski, Elaine C.
AU - Zhang, Ke Anne
AU - Motz, Benjamin A.
N1 - Publisher Copyright:
© The Author(s) 2021.
PY - 2021
Y1 - 2021
N2 - Psychology researchers have long attempted to identify educational practices that improve student learning. However, experimental research on these practices is often conducted in laboratory contexts or in a single course, which threatens the external validity of the results. In this article, we establish an experimental paradigm for evaluating the benefits of recommended practices across a variety of authentic educational contexts—a model we call ManyClasses. The core feature is that researchers examine the same research question and measure the same experimental effect across many classes spanning a range of topics, institutions, teacher implementations, and student populations. We report the first ManyClasses study, in which we examined how the timing of feedback on class assignments, either immediate or delayed by a few days, affected subsequent performance on class assessments. Across 38 classes, the overall estimate for the effect of feedback timing was 0.002 (95% highest density interval = [−0.05, 0.05]), which indicates that there was no effect of immediate feedback compared with delayed feedback on student learning that generalizes across classes. Furthermore, there were no credibly nonzero effects for 40 preregistered moderators related to class-level and student-level characteristics. Yet our results provide hints that in certain kinds of classes, which were undersampled in the current study, there may be modest advantages for delayed feedback. More broadly, these findings provide insights regarding the feasibility of conducting within-class randomized experiments across a range of naturally occurring learning environments.
AB - Psychology researchers have long attempted to identify educational practices that improve student learning. However, experimental research on these practices is often conducted in laboratory contexts or in a single course, which threatens the external validity of the results. In this article, we establish an experimental paradigm for evaluating the benefits of recommended practices across a variety of authentic educational contexts—a model we call ManyClasses. The core feature is that researchers examine the same research question and measure the same experimental effect across many classes spanning a range of topics, institutions, teacher implementations, and student populations. We report the first ManyClasses study, in which we examined how the timing of feedback on class assignments, either immediate or delayed by a few days, affected subsequent performance on class assessments. Across 38 classes, the overall estimate for the effect of feedback timing was 0.002 (95% highest density interval = [−0.05, 0.05]), which indicates that there was no effect of immediate feedback compared with delayed feedback on student learning that generalizes across classes. Furthermore, there were no credibly nonzero effects for 40 preregistered moderators related to class-level and student-level characteristics. Yet our results provide hints that in certain kinds of classes, which were undersampled in the current study, there may be modest advantages for delayed feedback. More broadly, these findings provide insights regarding the feasibility of conducting within-class randomized experiments across a range of naturally occurring learning environments.
KW - education
KW - evidence-based practices
KW - experiment
KW - feedback
KW - reproducibility
UR - http://www.scopus.com/inward/record.url?scp=85110363528&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85110363528&partnerID=8YFLogxK
U2 - 10.1177/25152459211027575
DO - 10.1177/25152459211027575
M3 - Article
AN - SCOPUS:85110363528
SN - 2515-2459
VL - 4
JO - Advances in Methods and Practices in Psychological Science
JF - Advances in Methods and Practices in Psychological Science
IS - 3
ER -