Examining the Correspondence Between Teacher- and Observer-Report Treatment Integrity Measures

Bryce D. McLeod, Kevin S. Sutherland, Michael Broda, Kristen L. Granger, Jennifer Cecilione, Clayton R. Cook, Maureen A. Conroy, Patricia A. Snyder, Michael A. Southam-Gerow

Research output: Contribution to journalArticlepeer-review

Abstract

Teacher-reported measures of treatment integrity (the extent to which prescribed practices are delivered as intended by teachers) have the potential to support efforts to evaluate and implement evidence-based interventions in early childhood settings. However, self-report treatment integrity measures have shown poor correspondence with observer-report treatment integrity measures, raising questions about score validity. This paper reports on the development and initial evaluation of the score reliability and validity of the Treatment Integrity Measure for Early Childhood Settings Teacher Report (TIMECS-TR), which is designed to address limitations of previous self-report treatment integrity measures that may have contributed to low correspondence with observer-rated measures. The TIMECS-TR includes 24 items designed to represent practices found in evidence-based interventions delivered in early childhood settings that target child social, emotional, and behavioral skills, rather than adherence to practices found in a specific evidence-based intervention. Fifty-four teachers (92.6% female, 7.4% male; 61.1% White) completed the TIMECS-TR weekly for a total of 618 times (M = 6.79 per child; SD = 2.16; range 2 to 11) about the practices they delivered with 91 children (45.1% female, 54.9% male; M = 4.53 years old; SD = 45.1% Black) who were at risk for emotional and behavioral disorders. Analyses indicated that the TIMECS-TR items evidenced mild to moderate test–retest score reliability over one week. However, analyses did not support the convergent score validity of the TIMECS-TR items or scale with observational ratings of the same practices. Teachers reported higher levels of practice delivery on the TIMECS-TR items relative to observer report. Overall, our findings raise concerns about the accuracy of teacher-report adherence measures. Lessons from this research can be used to identify possible reasons for the low correspondence between teacher- and observer-report treatment integrity measures so that future research can strive to dependably capture teacher delivery of the practices found in evidence-based interventions.

Original languageEnglish (US)
JournalSchool Mental Health
DOIs
StateAccepted/In press - 2021

Bibliographical note

Funding Information:
Preparation of this article was supported in part by a grant from the Institute of Education Sciences (R305A140487; McLeod & Sutherland).

Publisher Copyright:
© 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.

Keywords

  • Early childhood
  • Practice elements
  • Teacher implementation
  • Treatment integrity

Fingerprint Dive into the research topics of 'Examining the Correspondence Between Teacher- and Observer-Report Treatment Integrity Measures'. Together they form a unique fingerprint.

Cite this