Objective. To determine the interrater reliability and construct validity of 3 separate assessment tools for assessing trainee skills in pediatric airway endoscopy simulation. Design. An Objective Structured Assessment of Technical Skills (OSATS) was developed in which examinees were asked to name and assemble the airway foreign body instruments and retrieve a foreign body from an infant airway mannequin. Each examinee's performance was assessed in a blinded fashion by 3 pediatric otolaryngology faculty at separate residency programs using 3 assessment tools: (1) objective quantifiable measures list (eg, assists needed, forceps openings, foreign body drops), (2) 15-point OSATS checklist, and (3) Global Rating Index for Technical Skills (GRITS). Setting. Otolaryngology residency program. Subjects. Examinees (medical students, n = 3; otolaryngology residents, n = 17; pediatric otolaryngology faculty, n = 3) and raters (n = 3). Main Outcome Measures. Interrater reliability and construct validity. Results. Anonymized split-screen videos simultaneously capturing each examinee's instrument handling and the endoscopic videos were created for all 23 examinees. Nineteen videos were chosen for review by 3 raters. The interrater reliability as measured by the intraclass correlation for objective quantifiable measures ranged from 0.46 to 0.98. The intraclass correlation coefficient was 0.95 for the 15-point OSATS checklist and 0.95 for the GRITS; both showed a high degree of construct validity with scores correlating with previous experience. Conclusion. Assessment tools for skills assessments must have high interrater reliability and construct validity. When assessing trainee skills in pediatric airway foreign body scenarios, the 15-point OSATS checklist developed by this group or the GRITS meets these criteria.
|Original language||English (US)|
|Number of pages||5|
|Journal||Otolaryngology - Head and Neck Surgery (United States)|
|State||Published - Dec 2012|
Bibliographical noteFunding Information:
Competing interests: Some equipment used in this course was temporarily loaned to the authors at no charge by Karl Storz Endoscopy—USA. Robert M. Sweet is owner and employee for Red Llama, Inc and received grants from Olympus, Medtronic, and AMS. James D. Sidman is a consultant for Medtronic.
Funding source: This study was funded in part by a 2009 Resident Research Grant from the American Academy of Otolaryngology–Head & Neck Surgery Foundation ($10,000) and a University of Minnesota SimPORTAL Support Services Grant ($5744).
- Airway foreign body
- Competency assessment
- Resident education