Estimates of standardized mortality, hospitalization, and transfusion ratios are accompanied by uncertainty, as expressed by corresponding 95 percent confidence intervals.
One way to account for uncertainty is to take each standardized outcome measure and simulate an alternative value, by taking a random draw from a normal distribution with mean equal to the estimate of the measure and standard deviation equal to the standard error implied by the confidence interval.
Here we use public data from the July 2014 release of Dialysis Facility Compare to compile 5-star ratings according to CMS methodology and likewise according to an alternative methodology in which the uncertainty of standardized outcome measures is accounted, but weights assigned to all three domains and the seven constituent metrics are left unchanged.
Ratings assigned by the contrasting approaches are generally similar, but, nonetheless, discrepant ratings due to nothing other than statistical variation are apparent.
In total, more than 20 percent of facilities are assigned unequal numbers of stars by the contrasting approaches, a clear indication that star ratings with no mention of uncertainty are inappropriate for release to the public.
Although this criticism might be dismissed by claims that ratings with no mention of uncertainty are nonetheless “best estimates” of ratings, it is likely to be confusing to patients when facility ratings oscillate from year to year for no apparent reason.
Comparison of rating from CMS methodology vs. rating that reflects uncertainty in estimates of standardized outcome ratios
Number of facilities
(Alternative facility score in rows; CMS rating in columns)
Movement from CMS rating
(Movement in rows; CMS rating in columns)