Log in to MyACC
The Impact of Adjusting for Reliability on Hospital Quality Rankings in Vascular Surgery
Study Questions:
What effect would adjusting hospital mortality for statistical reliability have on hospital quality rankings in vascular surgery?
Methods:
The 2007 National Surgical Quality Improvement Project (NSQIP) database was queried for open and endovascular abdominal aortic aneurysm repair, carotid endarterectomy, lower extremity bypass, and aorto-femoral bypass procedures. A total of 14,559 of these operations were identified, and observed to expected (O-E) mortality ratios were calculated using standard NSQIP techniques. Bayesian statistical methods were used to adjust these estimates for statistical noise. Hospital rankings in vascular surgery based on the standard O-E ratios were compared to those rankings based on estimates that were adjusted for statistical noise.
Results:
In the total of 174 hospitals included in the 2007 NSQIP database, the average adjusted mortality rate was 2.4%, varying from 0% to 17%. After adjusting for statistical noise using the reliability adjustment, overall hospital mortality was reduced, varying from 1.7% to 4.1%. The proportion of mortality that could be attributed to statistical noise ranged from 94% in the lowest volume hospitals to 64% in the highest volume hospitals. Conversely, the proportion of “signal” of fraction of mortality that might be attributable to quality was estimated to range from 3% in low-volume to 18% in high-volume hospitals. When hospitals were ranked into quartiles of quality using traditional methods, quality adjustment resulted in reclassification of 43% as low quality or high quality. Specifically, among hospitals defined as highest quality, a full 51% were reclassified; 26% of hospitals classified as lowest quality were similarly reclassified.
Conclusions:
Adjusting mortality rates in vascular surgery for reliability reduces statistical noise and likely results in more accurate and stable estimates of hospital quality. Statistical adjustment for reliability should be standard for comparing hospital quality.
Perspective:
The authors clearly demonstrate the staggering amount of statistical noise in standard NSQIP statistical analysis used to determine the O-E ratios of outcome measures and its effect on hospital ranking. In an age of 'pay for performance' and 'selective referral practices,' the actual statistical methods used in measuring the patient outcome of interest has become increasingly important. This study not only makes an excellent argument for the use of their Bayesian statistical methods of measuring and then controlling for the effect of statistical noise, but also for the use of a composite measure using multiple measures in determining hospital performance and ranking.
Keywords: Outcome Assessment, Health Care, Quality Improvement, Hospital Mortality, Endarterectomy, Carotid, Lower Extremity, Noise, Aortic Aneurysm, Abdominal, Vascular Surgical Procedures
< Back to Listings
