MANAGED CARE June 2006. ©MediMedia USA
An exercise in "point-counterpoint" at a recent disease management meeting showed dramatic swings in views
*Four members of the National Managed Health Care Congress (NMHCC) Disease Management Outcomes Workgroup debated the pros and cons of six different issues before a group of knowledgeable listeners, and discovered that even in a short period, opinions could change significantly. Six issue statements were presented to the participants who were asked to respond to the statements with "agree, somewhat agree, neutral, somewhat disagree, and disagree" using wireless keypads. (The issue statements are below the graph.)
Three minutes
After the tally for one issue was displayed, one panelist spoke three minutes in favor of the statement. A second panelist spoke for three minutes against the statement. Then the poll was repeated. Dramatic shifts were evident in the responses to half of the statements. This indicates that discussion and debate is a valuable tool for the DM industry.
Jeff Gruen, MD, MBA, had assembled the DM Outcomes Workgroup for an earlier NMHCC conference to help attendees improve their comprehension of the complex financial and methodological elements of impact assessment in DM programs.
The workgroup has now presented successfully seven times at National Managed Health Care Congress/Disease Management trade shows, has produced a publication based on these results (Joint Commission Journal on Quality and Safety. November 2004, Volume 30, Number 11, Page 616. "Assessing Return on Investment of Defined-Population Disease Management Interventions." Authors: Thomas W. Wilson; Jeff Gruen; William Thar; Donald Fetterolf; Minalkumar Patel; Richard G. Popiel; Al Lewis; David B. Nash), and continues its work.
At the September 2005 Disease Management Congress in Orlando, Fla., the workgroup conducted a session titled "Who Wants to Avoid the Millionth Error? DM Reality Show Explores Issues Driving the Outcomes Measurement Debate."
Before each debate began, the question was read to audience members who were given the choice of five answers. The audience had about 30 seconds to respond, and the sums were displayed immediately.
One panelist defended the statement for three minutes (the pro position), followed by a three-minute speech from one panelist who argued against the issue (the con position). The poll was repeated for each issue, with the post-debate sums and pre-debate sums displayed.
Point-counterpoint
Responses of "agree" and "somewhat agree" were combined into one category and "somewhat disagree" and "disagree" into another. Statistical significance tests were performed after the debate.
On 5 of the 6 issues debated, the percent that strongly agreed or agreed dropped as a result of the debate. Half of the time, the percent who strongly agreed or agreed showed a statistically significant difference after the discussion.
The three issues that show the most changes in participants' opinions were, in descending order of change: 1) uniform ROI measurement standard needed (number 1 in chart below); 2) controls regression-to-the-mean (number 4); and 3) DM dies in 3–5 years if federal projects fail to find value (number 6).
General conclusions from those polled, to the degree that they represented their respective institutions and the industry, would include the following:
- Following this nonscientific market poll, the disease management industry should certainly consider the creation of a single universal ROI methodology, as 58 percent agreed or somewhat agreed with that statement after the debate. However, the industry may want to carefully think over how best to proceed on this issue, as prior to the debate a full 81 percent agreed or somewhat agreed with this statement. This was the biggest shift in opinion that was seen in the six questions. This shift in the audience's outlook after six minutes of discussion suggests that this effort should proceed carefully.
- Disease management program studies that have been published are overwhelmingly believed to not include sufficient methodological adjustments to establish equivalence between the reference group and the DM intervention group. The debate did not appreciably change the opinion on this issue. This is a potentially challenging problem as most studies conducted in DM are not randomized control trials. The randomized control trial study design is most likely to lead to equivalent reference groups, but there is no guarantee that randomized control trials will achieve equivalence.
- Participants in the meeting strongly felt that disease management outcomes are not primarily a result of reduction in health care provider practice variation, but rather, the change in patient behavior. This position was improved by the debate, with 9 percent of the audience shifting position toward patient change as the primary driver.
- The problem of adequately controlling for regression to the mean (RTTM) was identified as the most serious negative of all the issues. After the debate, 86 percent of the participants — up from 62 percent prior to the debate — did not believe that DM had adequately controlled for this phenomenon. Simply put, RTTM refers to the observed decline in costs and utilizations in many DM populations when using a pre-intervention/post-intervention study design. But the decline seems to occur even when DM is not present, suggesting that the programs are not responsible for the change, but that it is a natural phenomenon.
- For DM initiatives, the belief that a ROI of only 1:1 (i.e., where DM costs = DM benefits) was consistent (41 percent to 38 percent, not statistically significant). Most importantly, over 50 percent believed that this statement was not supported, presumably because evidence supports the contention that DM ROIs are greater than 1:1. In other words, the audience believed that DM did, in fact, have a positive ROI in both published and nonpublished studies.
- The conference participants were initially skeptical regarding the future of the DM industry, given a hypothetical unfavorable Medicare Chronic Care Improvement Project (CCIP) evaluation (now called the Medicare Health Support project). However, the audience changed its position after the debate, most likely because of the recognition by the con position that disease management offers a variety of multidimensional contributions, including clinical improvement, patient satisfaction, productivity improvement, and other softer outcomes that result in overall better health. The results from DM are not and should not be single-dimensional.
Authors' commentary
We believe that the audience informed us that the industry must move toward more standardized methodologies that can more adequately adjust for regression to the mean and nonequivalence between the DM intervention population and the reference (e.g. trend, pre-period, concurrent period, nonparticipants, classic control group, etc.). We agree with this sentiment.
The fact that there were significant changes in opinions in one half of the issues leads us to believe that there is a great need for civil, structured discussion and honest and respectful public debate of the important issues facing DM. These discussions should take place at conferences, at other meetings, and in the pages of magazines.
Given the issues in the peer-reviewed literature, we encourage more space be devoted in peer-reviewed journals for letters to the editor. This would constitute an on-going public peer review of peer-reviewed literature.
A similar debate will be conducted at the Health Management Congress in the fall.