Hospitals' Self-Reports Skew Results on Grading System, Study Shows

New Leapfrog methodology and grades due soon

A variety of organizations combine different measurements to produce “report cards” showing how well hospitals serve their patients and keep them safe. Some use only objective patient outcomes, such as infection rates, while others allow hospitals to report on their efforts to improve safety. But a new study finds that some methods may put too much weight on the wrong things.

The study, published in the journal Medical Care by a team at the University of Michigan (UM), takes an in-depth look at the Hospital Safety Grade system, run by the nonprofit Leapfrog Group. The researchers found that most hospitals that participate in Leapfrog's Hospital Survey self-report that they're in perfect compliance with most, if not all, of the "Safe Practices" that the survey asks about. But the researchers show that how a hospital did on these measures had little in common with how that same hospital did on independent measurements of hospital-acquired infections—or with whether the government had penalized it for high infection or readmission rates.

"The Safe Practices part of the Hospital Safety Grade is based on whether a hospital self-reports that it has adopted certain protocols, and not whether those protocols are actually followed," says Jennifer Meddings, MD, MSc, the study's senior author and an assistant professor of internal medicine at the UM Medical School. For example, she says, one Safe Practices measure involves hand hygiene. "It may be news to consumers that this measure involves hospital self reports of having specific hand washing protocols in place, not how often health care workers are washing their hands."

She also notes that any one hospital's Safe Practices scores count toward its overall Hospital Safety Grade based on a comparison with the average Safe Practices measure score reported across all participating hospitals. So, hospitals receive little grade advantage by reporting high scores—but are at risk of much lower grades if they report less-than-perfect scores.

The researchers note that Leapfrog will issue a new grading formula next month, but not in time to calculate the new grades that will come out soon after that.

Hospital Safety Grades combine data from a number of national sources, including public data from the Centers for Medicare and Medicaid Services and the Centers for Disease Control and Prevention—and for hospitals that choose to participate, the voluntary Leapfrog Hospital Survey.

Just over half of hospitals in the study completed the Leapfrog survey, which includes self-reporting of eight National Quality Forum Safe Practices measures. If a hospital reports on them, it accounts for nearly a quarter of the hospital's total grade. Hospitals that do not complete the voluntary Leapfrog Hospital Survey also receive a Leapfrog Hospital Safety Grade, calculated entirely using publicly available measures.

The researchers ran simulations to see how these nonparticipating hospitals' grades might have changed if they had self-reported low, average, or perfect Safe Practices measures, while their infection rates remained the same. They found that—even if their actual performance on hospital infections is good—such hospitals would likely end up with lower Leapfrog Hospital Safety Grades if their performance on the self-reported Safe Practices was anything less than perfect. In other words, hospitals that self-report less-than-perfect Safe Practices scores are likely better off not completing the Leapfrog Hospital Survey.

More than half of hospitals that complete the Leapfrog survey report perfect performance on all of the Safe Practice measures. The researchers say it's time to put more weight on publicly reported safety measures from objective sources rather than self-reporting of protocols.

Source: University of Michigan; March 28, 2017.