Evaluating A New Technique For Improving The Reliability Of Surgical Outcomes Measurement

Introduction: Reliability adjustment, a method for better accounting for statistical “noise”, is becoming more widely used outside surgery. We sought to determine the impact of this technique in surgery. Methods: We used the American College of Surgeons - National Surgical Quality Improvement Program (ACS-NSQIP) to identify all patients undergoing colon resection in 2007 (n=179 hospitals, n=16,262 patients). We first used standard methods to generate risk-adjusted mortality for each hospital. We then used empirical Bayes techniques to adjust for reliability. Hospitals were grouped into five equal-sized groups (quintiles). To evaluate the impact of reliability adjustment, we estimated the extent to which hospital-level variation was reduced and compared quintile rankings with and without reliability adjustment. Results: Adjustment for reliability greatly diminished the apparent variation in risk-adjusted mortality across hospitals. Before adjustment for reliability, risk-adjusted mortality varied nearly 5-fold (1.8% to 8.4%) from the best (top 20%) to worst (bottom 20%) quintiles. After adjusting for reliability, the mortality rates varied less than 2-fold (3.3% to 5.7%) from the best to worst quintiles. Reliability adjustment resulted in large changes in hospital quintile rankings. Of the hospitals in the top 20% using standard methods, only 45% (20/44) remain in the top 20% after reliability adjustment. Likewise, of the hospitals in the bottom 20% using standard methods, only 55% (25/45) remained in the bottom 20% after reliability adjustment. Conclusions: Reliability adjustment results in large changes in hospital performance for colon resection. This technique provides more accurate estimates of hospital performance and should be considered in the place of standard methods.