Are quality assessment methods any good?

A systematic review can be viewed as a study of studies,¹ done to understand why different studies of a given treatment gave varying results. One explanationie, one "independent variable" potentially explaining variation in study results-is the studies' methodological quality, defined for the present purpose as internal validity, the extent to which potential for systematic error (bias) is minimized by careful study design, conduct, and analysis. Two broad approaches have been proposed to assess study quality and to construct such independent variables: the components approach, in which specific aspects of design, execution, and analysis are judged as adequate or inadequate; and quality scales, in which the components are given numerical scores, which are then summed to give a composite score. There is considerable empirical evidence about the association between study results and specific components of quality. Most commonly (but not always), lower quality on a component is associated with a larger estimated effect for a new treatment. Evidence regarding quality scales is much thinner, showing no association with study results except for the scale of Jadad et al.² This article concludes that assessing study quality is markedly hampered by poor reporting in journal articles; that summary scores from quality scales are inherently problematic; but that nonetheless a case can be made for routinely assessing certain components of study quality.