A Preliminary Investigation of the Empirical Validity of Study Quality Appraisal

When classifying the evidence base of practices, special education scholars typically appraise study quality to identify and exclude from consideration in their reviews unacceptable-quality studies that are likely biased and might bias review findings if included. However, study quality appraisals used in the process of identifying evidence-based practices for students with learning and other disabilities have not been empirically validated (e.g., studies classified as unacceptable quality shown to have different, and presumably more biased, effects than high-quality studies). Using Gersten et al.’s (2005) approach for appraising the quality of group experimental studies in special education, we examined whether (a) studies classified as unacceptable quality and high quality had meaningfully different effects and (b) unacceptable-quality studies were more likely to have outlying effects than high-quality studies among 36 group experimental studies that investigated the effectiveness of instructional practices for students with learning disabilities. Our preliminary analyses found that the effects of unacceptable-quality studies were not meaningfully different from the effects of high-quality studies. We discuss implications of these findings and call for more research to be conducted in this area.