How much do differences in medical schools influence student performance? A longitudinal study employing hierarchical linear modeling

Background: Medical school curricula have undergone considerable change in the past half century. There is little evidence, however, for the impact of various curricula and educational policies on student learning once incoming performance and the nonrandom nature of students nested within schools has been accounted for. Purpose: To investigate effects of school variables on United States Medical Licensing Examination (USMLE) Step 1-3 scores over an 11-year period (1994-2004). Methods: Using Association of American Medical Colleges and USMLE longitudinal data for 116 medical schools, hierarchical linear modeling was used to study the effects of school variables on Step 1-3. Results: Mean unadjusted between school variance was 14.74%, 10.50%, and 11.25%, for USMLE Step 1-3. When student covariates were included, between-school variation was less than 5%. The proportion of variance accounted for in-student-level performance by the covariates ranged from 27.58% to 36.51% for Step 1, 16.37% to 24.48% for Step 2, and 19.22% to 25.32% for Step 3. The variance accounted for in student performance by the student covariates ranged between 81.22% and 88.26% for Step 1, 48.44% and 79.77% for Step 2, and 68.41% and 80.78% for Step 3. School-level variables did not consistently predict for adjusted mean school Step performance. Conclusions: Individual student differences account for most of the variation in USMLE performance with small contributions from between-school variation and even smaller contribution from curriculum and educational policies.