Novel methods to evaluate fracture risk models

Fracture prediction models help to identify individuals at high risk who may benefit from treatment. Area under the curve (AUC) is used to compare prediction models. However, the AUC has limitations and may miss important differences between models. Novel reclassification methods quantify how accurately models classify patients who benefit from treatment and the proportion of patients above/below treatment thresholds. We applied two reclassification methods, using the National Osteoporosis Foundation (NOF) treatment thresholds, to compare two risk models: femoral neck bone mineral density (BMD) and age (simple model) and FRAX (FRAX model). The Pepe method classifies based on case/noncase status and examines the proportion of each above and below thresholds. The Cook method examines fracture rates above and below thresholds. We applied these to the Study of Osteoporotic Fractures (SOF). There were 6036 (1037 fractures) and 6232 (389 fractures) participants with complete data for major osteoporotic and hip fracture, respectively. Both models for major osteoporotic fracture (0.68 versus 0.69) and hip fracture (0.75 versus 0.76) had similar AUCs. In contrast, using reclassification methods, each model classified a substantial number of women differently. Using the Pepe method, the FRAX model (versus the simple model) missed treating 70 (7%) cases of major osteoporotic fracture but avoided treating 285 (6%) noncases. For hip fracture, the FRAX model missed treating 31 (8%) cases but avoided treating 1026 (18%) noncases. The Cook method (both models, both fracture outcomes) had similar fracture rates above/below the treatment thresholds. Compared with the AUC, new methods provide more detailed information about how models classify patients.