The Best ROI Meta-Analysis Ever: Case Closed?

We’ve known Michael O’Donnell, the publisher and editor in chief of the American Journal of Health Promotion, for 30 years. He is not prone to hyperbole. So he got our attention when he recently wrote that he had just published “the most extensive and well-conceived review conducted to date” on the financial impact of workplace health promotion.

He calls the paper “one of the best reviews ever conducted on any topic in workplace health promotion.” The study by Siyan Baxter and colleagues shows how the quality of the research methodologies used in the 51 interventions examined in their final analysis affected the magnitudes of the ROI’s reported. An archived webinar that summarizes the study findings is available free on this page at the  American Journal of Health Promotion.

Studies vary

There is a time-honored belief among health services researchers that the more rigorous the methodology, the smaller the size of the differences between groups will be. This is precisely what Baxter and colleagues found with respect to return on investment, which they calculated as ROI = (benefits – program costs) / program costs rather than the more typical approach of reporting ROI as the ratio of benefits to costs, i.e., ROI = benefits : costs. Nevertheless, Baxter, who is a PhD candidate in medical research at Australia’s University of Tasmania, found ROIs averaging 0.26 ($1.26 per dollar invested) in the 18 high-quality studies and –0.22 ($0.78 per dollar invested) in 12 studies using the most rigorous methodology, namely, the randomized controlled trial (RCT). For the 43 less rigorous studies of moderate or low quality, Baxter found ROIs averaging 1.79 (i.e., benefits of $2.79 per dollar invested), which is consistent with the oft cited ROI of 3:1 reported in a recent review by Harvard Economist Katherine Baicker and her colleagues.

And, similar to the results reported by StayWell researchers in Population Health Management, Baxter found that it took up to five years to garner significant ROI results. In the Staywell study, we also used more stringent selection criteria than most ROI reviews and found what we expected: more scientific rigor nets lesser, albeit still positive, ROI averaging about $1.87 in benefits per dollar invested.

Agree and disagree

We agree with those who question ROI estimates based on averaging the results of studies regardless of quality or rigor, which this new review shows leads to unrealistic expectations about the financial impact that worksite health promotion programs can deliver.

However, we disagree with those who dismiss all positive ROI findings based on the shortcomings of lower quality, less rigorous studies.

Can poorly conducted research yield unexpectedly positive findings? Certainly. Do piecemeal health promotion programs that are overly prescriptive or use inordinate financial incentives or unnecessary testing produce disappointing results? Probably.

What the Baxter review shows, however, is that well-designed, rigorously evaluated worksite health promotion programs can yield positive net financial benefits. This is the same conclusion reached by many experts in the field based on carefully weighing the same full body of peer-reviewed evidence included in the Baxter review.

We expect the Baxter study to join the Baicker meta-analysis as one of the most cited in the field of worksite wellness by those interested in a serious examination of what is both possible and realistic when it comes to ROI.

Program quality

To paraphrase O’Donnell’s take-away, well-designed wellness programs seem to pay for themselves, but the reported magnitude of the financial return depends on the quality of the program and the rigor of the evaluation design. We expect the case is far from closed for a few bloggers, given their propensity to ignore best-practice findings to fuel an artificial controversy.

We’ll save our angst for reviews related to carbohydrates, aspirin, and barefoot running. Unlike wellness ROI, where the evidence from best practice programs is increasingly clear, the science in these complex areas has truly been controversial.