MANAGED CARE February 1998. ©1998 Stezzi Communications
TIME FOR A CLOSER LOOK AT COST-EFFECTIVENESS?
A doctor trained in informatics argues that managed care must borrow the principles of accounting to appraise more realistically the costs of medical interventions.
To point out to health care decision-makers today that they ought to consider cost as well as clinical efficacy would risk insulting them. The whole managed care revolution — not to mention the magazine you hold in your hands — reflects acknowledgment of the need to spend resources wisely. But a widely noticed article ("Interpreting Cost Analyses of Clinical Interventions," Journal of the American Medical Association, Jan. 7, 1998) brought readers up short with an almost equally basic suggestion. Politely but firmly, the article contended that a great many of the cost-effectiveness calculations presented in clinical studies over the past few years have been just so much hooey. While the randomized controlled trial has gained acceptance as a kind of "gold standard" for clinical efficacy over the years, it said, there is still no comparable standard for cost-effectiveness. Discussions of cost that do appear, the article went on, suffer from varied, conflicting and sometimes too-narrow definitions, and they chronically omit or underreport startup costs and general overhead. In short, it concluded, at a time when physicians and managed care executives need desperately to weigh possible interventions in terms of cost-effectiveness, "Economic analyses currently available are often incomplete and fail to provide relevant cost information to practitioners.... The lack of data and inconsistencies of method should prompt researchers to supplement clinical trial checklists with a costing protocol."
The author was E. Andrew Balas, M.D. Originally from Budapest, he came to this country in the 1980s and has supplemented his medical education with an M.S. degree in applied mathematics and a Ph.D. in medical informatics. He is on the faculty of the Health Services Management Program at the University of Missouri at Columbia, and serves in the training program at the National Library of Medicine and the National Center for Managed Health Care Administration. Managed Care sought out Balas recently for an interview.
MANAGED CARE: Your article in JAMA reminded me of the electric car. Its supporters often call it completely pollution-free, while backers of natural gas vehicles point out that the electricity it uses must come from somewhere, and that is part of its environmental costs. It seems you are telling people in health care that they have been neglecting part of the cost picture all along, and that in the process they've distorted some of the cost-effectiveness discussions they have been making. Is that a correct read?
BALAS: Yes. Randomized controlled clinical trials are considered the top-quality source of evidence on the clinical impact of an intervention. And we have very good trials in many areas — not just testing new drugs, but also testing surgical procedures and all kinds of other interventions, like patient education or computerized information services, or home care vs. hospital care. So we have a fairly good idea about the clinical impact of these interventions. But the problem is that the cost is not analyzed correspondingly. Or if it is analyzed, it is very inconsistent and very vague in the literature. Managed care's charge is to control costs. How can you do it rationally when there is no way to make a rational decision because you don't know how much you lose on the outcome when you cut the cost?
MC: If you don't have accurate cost information, clearly you can't make accurate determinations.
BALAS: Correct. And you know, there are all kinds of studies that analyze costs. But we do not just need to talk about costs. We have to be in a situation where cost can be linked to the outcome-effective intervention, so you know what you're paying for. That's a critical piece of information, and it's obviously missing.
MC: I gather that not only are we lacking this information, but to some degree we are under a false impression that we have it.
BALAS: That is absolutely correct. We have found that nearly half of the articles that make statements about cost-effectiveness do not present any numbers to substantiate their claims. And it is very difficult to trust these statements when they are presented without numbers. The other half present numbers, but they're very inconsistently collected, and it is quite obvious that most of them represent an underestimate of the actual costs, because major cost factors are ignored.
MC: Is anybody doing it in a more realistic way? Any HMO or company or hospital you see as actually appraising costs accurately?
BALAS: That is an interesting question. Company studies are usually for internal consumption. We analyze the research base, what is published in the scientific literature. You know, you can do all kinds of good studies in a managed care or hospital environment. But it is very unlikely that people are able to do those studies because it's a quite extensive and expensive research venture. It's in everybody's interest to make published research results relevant to the daily decisions of the medical care directors and managed care executives.
MC: Are you implying that research that goes on behind the closed doors of profit-making corporations that are watching every dime might be a little more economically realistic?
BALAS: Actually, I do not believe that. I don't believe there are secret, unpublished studies being done everywhere in the country that are far better from what we have in the literature. Most likely they are just about the same or maybe a little worse. Many of these studies are adequately funded by the National Institutes of Health or major private foundations like the Robert Wood Johnson Foundation. And these are not cheap studies by any means. The costs of these studies begin at $40,000 to $50,000 and go up to the several-million-dollar range.
MC: What inspired this line of research for you?
BALAS: Our research agenda is really the collection or processing of the best available evidence, because we all agree that not everything in the medical literature represents the same value. There are some very good studies and there are some bad studies, miserably poorly conducted or unreliable studies. When you do a literature search, you get everything. Most experts would agree that the majority of retrieved articles will be probably in the lower quality range for various reasons. The high-quality evidence is there and we collect it. We were interested in the practical information and message of these articles. We published many papers on the clinical impact consequences of these studies, and this particular study focused on cost as it relates to quality.
MC: It sounds as if you're suggesting that there is excessive publication going on.
BALAS: If you look at the contemporary evidence rating systems, they put the randomized controlled clinical trial on the top and the expert opinion and the consensus statement on the bottom. If you do a literature search about a certain professional issue, most likely at least half of the retrieved articles will come from the expert opinion category.
MC: Do you think there is too much being published?
BALAS: It's hard to say. If only we could increase our own efficiency by publishing better studies and fewer opinions. But research publications have multiple roles, and to present new scientific evidence is only one of their roles. Another role is to educate people, and opinion papers usually are more targeted towards that.
MC: What is the most common mistake HMOs make when determining that one intervention or treatment is more cost-effective than another?
BALAS: One of the most prevalent mistakes-- and it's not necessarily their sloppy decision-making process, it's just a lack of evidence — is that they rely on incomplete cost information when making those decisions. And that means that some components are usually ignored, such as startup costs or overhead expenses associated with clinical interventions.
MC: Such observers as Robert Brook, M.D., of the Rand Corporation have said for years that we do one-third or one-fourth more medical interventions then we really need to, but it's a question of finding which one-fourth that is. We overtreat. If we had truly accurate cost information, would that help the problem of overtreatment and limit interventions to the strictly necessary and efficacious ones?
BALAS: It is an intertwined problem, because when we are talking about unnecessary treatment, then primarily we are talking about quality — a clinical procedure that is done, but that fails to result in improved health status. There are many procedures that are used but that are not necessarily of benefit. It's a very complex problem why these happen.
MC: Sometimes, I understand, they've been statistically correlated not with any constant need in the population, but with the prevalence of the specialists or facilities offering the intervention.
BALAS: Yes, that is one of the factors that seem to apply. And our lives would be much easier if we had top-quality evidence on all procedures. In reality, a very small percentage of clinical procedures have ever been tested in controlled clinical experiments. There are a lot of speculative procedures. In some cases they work well, but in others the speculation is misleading. On one of our tables we highlight a study that was significantly better than the average, and it can illustrate how the costing protocol that we recommend can be used. And that is a message not just for the researcher, but also for the practitioner. Whenever you see an article making a statement regarding cost-effectiveness, put the cost data into that protocol. And when you find empty cells, think about whether they are zeroes or just unpublished or ignored costs.
MC: By empty cells, you mean parts of the equation for which no information is presented?
BALAS: Right. In the table you will find that there are certain lines that will not be filled with numbers, because most publications do not do that. So that will help you to realize that, yes, it may be very cost-effective if you restrict your attention to the operating expenses, but when you think about the startup costs it may not be that appealing.
MC: If I understand your paper correctly, there were 181 trial reports that were eligible at a certain midpoint in the process of elimination, having showed up as mentioning "cost," "cost-effective" or related terms on a Medline search from 1966 to 1995. Of those 181, almost exactly half were since 1991. Right?
BALAS: Right.
MC: That tells us something right there, does it not? That cost has mushroomed as a concern?
BALAS: Yes. Probably we can make that statement. But it's hard to separate that from the fact that clinical trials fortunately are becoming more prevalent in the medical literature.
MC: But the heightened interest in cost considerations has not been accompanied by any improvement in the accuracy of calculating costs over the last 30 years?
BALAS: That's absolutely correct, and that's a key observation, because what we know from our previous studies, and from other studies too, is that clinical trial methodologies are improving. The average trial today is much more reliable than a trial from five, 10, 15 or 20 years ago. That is a very good trend. But if you look at the cost calculation side, there is absolutely no improvement over time. We need a change.
MC: Can you give me an example of a specific clinical choice among interventions that has been made in managed care based on faulty or incomplete cost calculations?
BALAS: That is a difficult question. But I can tell you that if the evidence is not there, then there is very little room for good decisions, and a good decision is like hitting the lucky cherries on a slot machine.
MC: It sounds as if you're saying this is a huge blind spot that limits the otherwise improving quality of randomized clinical trials and the reports thereof.
BALAS: That's true. And if you look at the evolution of our science, the progress is truly tremendous in terms of research methodologies. A couple of decades ago a case study was a sufficient substantiation of any kind of intervention. That's how people published articles. And at the end of last century they presented cases to prove that bloodletting could cure diabetes.
MC: Do research professionals need more training in economics?
BALAS: That is a very interesting question. I would say maybe, but the key issue is not there. The key issue is, are they able to collaborate with finance professionals? Because if they can learn it, that's OK, but it's a tremendous investment of their time. Why don't we cooperate? Why don't we talk to people who are familiar with cost calculation methodologies, experts in finance and accounting?
MC: Who is such an expert on the cost side?
BALAS: There are some on every campus, in the school of business or the finance department.
MC: But health economics has become pretty sophisticated, hasn't it? You seem to be suggesting that it's still in the dark when it comes to evaluating procedures.
BALAS: Health economics frequently has a macro perspective, and if you look at publications in the field there are frequently graphs with curves, but without numbers. They illustrate models and concepts. This is a different level of activity, closer to the accounting and finance area. We need to communicate with those people much better, and doctors can communicate. I have an advanced degree in mathematics, but I still talk to statisticians when I talk about analysis.
MC: It sounds as if you're saying that doctors in managed care must take the further step of seeking out a rigorous accounting expertise that business schools have long known but that has been a stranger to medical schools.
BALAS: That is absolutely correct. And communication is the key to successful managed care.
MC: What's the essential lesson of your work for the primary care physician in managed care today?
BALAS: Try to rely on the best available evidence whenever you make a policy decision, which means something for the longer run, something for a group of patients. And don't accept cost-benefit statements without numbers.
MC: What is your own health care coverage?
BALAS: It's an HMO.
MC: Has your personal experience been fairly successful?
BALAS: I think my personal experience was good. I would hesitate to generalize it to others, because being a physician always helps. But I think that managed care is able to give good care, and it's in an excellent position to reinforce and highlight preventive care. And that is very valuable. And that comes not so much from my personal experience, but from my research studies. Managed care plans that are encouraged to measure and disclose preventive care performance fare much better than other types of organizations.
MC: Are you happier with the way quality is now being measured by the National Committee for Quality Assurance, the Joint Commission on the Accreditation of Healthcare Organizations and others than you are with the way cost has been measured?
BALAS: Yes. The quality measurement is proceeding nicely. I think especially the NCQA Health Plan Employer Data and Information Set (HEDIS) is a very strong data set, with very carefully selected indicators. And the performance of the managed care industry is truly remarkable, because this is a major effort for each and every organization to pull the data together.
MC: And yet we have, in the media certainly and to some extent in the public, a backlash going on against managed care. One hears horror stories and a lot of griping and complaining.
BALAS: Yes, I know that, especially in connection with my health policy fellowship. You know, stories frequently drive our decision-making. On the other hand, managed care has been very good in observing and reporting its own clinical performance, while other parts of the market did not disclose anything. And that is not a fair comparison. What did we know about the diabetic eye exam rate in the old fee-for-service system? I have a few studies from the fee-for-service system, and I can tell you that the average of the managed care plans is much, much better.
MC: Thank you, Dr. Balas.