Concerns for the reliability and validity of the national stroke project stroke severity scale

Background: The National Stroke Project (NSP) was a retrospective cohort study of US Medicare beneficiaries hospitalized with stroke or transient ischemic attack (TIA). The NSP included a simple assessment of stroke severity (NSP-Stroke Scale, NSP-SS). Used for risk adjustment in outcome studies, the reliability and validity of the NSP-SS have not been assessed. We determined the reliability, concurrent and construct validity of theNSP-SS. Methods: The initial neurologic examinations of 100 consecutive patients hospitalized with ischemic stroke/TIA in a single academic medical center were reviewed. The NSP-SS was retrospectively scored twice by the same rater and independently by a second rater to assess reliability. The National Institutes of Health Stroke Scale (NIH-SS) was also scored retrospectively and used as the criterion standard for concurrent validity. Construct validity was based on discharge status. Results: The NSP-SS had moderate-substantial inter-rater (weighted kappa, κ _w = 0.66, 95% CI 0.55-0.77) and intra-rater (κ _w = 0.63, 95% CI 0.52-0.75) reliability. Correlation between NSP-SS and NIH-SS scores was moderate (Spearman r = 0.65, 95% CI 0.52-0.75, p < 0.0001) but some categorizations in the NSP-SS seemed inappropriate reflecting poor content validity. Each NSP-SS point was associated with a greater likelihood of poor outcome (OR = 2.1, 95% CI 1.1-3.7, p = 0.016). Based on dichotomized scores (NSP 0-2 and NIH-SS <6; mild deficits), the NSP-SS sensitivity was 70.9% (95% CI 57.9-81.2%), specificity 82.2% (95% CI 68.7-90.7%), likelihood ratio for severe stroke 4.0 (95% CI 2.1-7.6) and likelihood ratio for mild stroke 0.3 (95% CI 0.20-0.5). The dichotomized NSP-SS and NIH-SS similarly predicted poor outcome (NSP-SS >2, OR = 4.7, 95% CI 1.7-13.0, p = 0.003 vs. NIH-SS ≥6, OR = 4.4, 95% CI 1.5-13.0, p = 0.006) with excellent discrimination (C = 0.827 and 0.826, respectively). Conclusion: The NSP-SS has moderate-substantial reliability but poor content validity and poor to moderate concurrent validity as compared with the NIH-SS. In addition, it is not clear that the NSP-SS is easier to extract from medical records than the NIH-SS. Given this, and its other limitations, the utility of this scale for risk adjustment in future stroke outcome studies is questionable.