Study: Hospital Ratings May Rely on Bad Data

Rush University researchers cry ‘foul’

The best hospital rating systems in the United States––including the government-sponsored Hospital Compare, Consumer Reports, and Leapfrog––try to account for severity of illness. However, a recent analysis of the “Best Hospitals” rankings from U.S. News & World Report shows that these rating systems often depend on faulty data sets and performance metrics.

Rush University Medical Center consistently has scored high on patient safety measures, according to most rankings. But last year, the U.S. News “Best Hospitals” issue gave Rush the publication’s lowest ranking for patient safety (one out of five).

Suspecting a broader problem, a Rush team analyzed data from a sample of hospitals and discovered that Rush was not the only organization inaccurately ranked. The team’s findings were published in the October issue of the Joint Commission Journal on Quality and Patient Safety.

Two hospital characteristics tended to skew U.S. News patient safety scores: Taking care of large numbers of patients (i.e., high volumes) and admitting a high percentage of patients transferred from other hospitals.

The Rush investigators found that the U.S. News data set was missing key information, such as whether patients with pressure ulcers had the sores when they were admitted to Rush. If conditions were present before arrival, Rush should not have been penalized, the authors contend.

Rush notified U.S. News about its findings in 2015, and the journal made some revisions to its data methodology. As a result, Rush’s patient safety score increased to four out of a possible five in the 2016–2017 U.S. News ratings, released in August.

The Rush analysis of the U.S. News ranking system confirmed what many experts had suspected, according to the authors. “People have argued and theorized that if you take care of a lot of very sick people or have a lot of transfers, then the data could be biased,” said investigator Bala Hota, MD, PhD. “This study proves that. It shows that data are not perfect, and that misunderstandings in the data can negatively portray a hospital.”

Acknowledging that hospital rankings have room for improvement, U.S. News is convening a panel of experts in November to discuss how to improve evaluations of hospital quality, safety, and performance, according to a press release.

In the meantime, the Rush investigators said they will continue to measure, track, and share patient safety data.

Source: Rush University Medical Center; October 5, 2016.