Sue McQuillian, FSA
Consulting Actuary, Milliman USA
PDF version: 


Recent regulatory pressures and certification requirements have heightened the need for payer organizations to abide by specific standards regarding medical management operations. Payer organizations that are making medical-necessity determinations regarding reimbursement for health care services, as well as risk-bearing provider groups to whom some of these functions have been delegated, must insure consistency and appropriateness in their determinations. The most powerful accreditation body for health care payers, the National Committee for Quality Assurance (NCQA), requires that payer organizations carry out periodic inter-rater reliability assessments to insure consistency in medical management decision making.

An inter-rater reliability assessment or study is a performance-measurement tool involving a comparison of responses for a control group (i.e., the "raters") with a standard. Inter-rater reliability (also called inter-observer reliability) traditionally refers to how well two or more raters agree and is derived from the correlation of different raters' judgments. For the purposes of this paper, inter-rater reliability is a measurement of how well raters agree with a standard, which is more of an assessment of the validity of the responses. The purpose of the study is to determine whether the raters have been consistently trained and are applying that training in a consistent fashion. The analysis is intended to gauge the raters' observations and reactions resulting from a specific situation. The principles discussed herein would apply to any set of utilization management guidelines.

An inter-rater reliability assessment can be used to measure the level of consistency among a plan or provider group's utilization management staff and adherence to organizational medical management criteria or standards. Reasons for conducting an inter-rater reliability study within an organization include:

  • Minimizing variation in the application of clinical guidelines;
  • Evaluating staff's ability to identify potentially avoidable utilization;
  • Evaluating staff's ability to identify quality-of-care issues;
  • Targeting specific areas most in need of improvement;
  • Targeting staff needing additional training; and
  • Avoiding litigation due to inconsistently applied guidelines.

NCQA requires that health plans develop and implement an inter-rater reliability process for Health Plan Employer Data and Information Set (HEDIS) compliance. NCQA is an independent review organization dedicated to evaluating and reporting on the quality of managed care organizations. HEDIS is a set of standardized performance measures developed by NCQA with assistance from managed care organizations and employers concerned with quality health care. The performance indicators in HEDIS are continually developing, but most involve measuring access to care, health plan service, provider qualifications, activities that assist people to recover from illness, and management of chronic illness. The combination of these measures is intended to provide a tool for performance comparison of different health plans.

An inter-rater reliability study can provide measurement of many of these quality indicators. It can most readily be used to measure access, as NCQA looks for "fair and consistent health plan decisions about medical treatments and services provided to plan members." It also can be used to measure service, as NCQA looks for "actual improvements that the plan has made in care and service" ("What NCQA Looks for in a Health Plan," «»). This latter indicator can be measured by reviewing the results of the inter-rater reliability study from year to year.

An inter-rater reliability assessment can be a useful tool for a health plan or provider organization. However, as with any benchmarking exercise, it cannot in and of itself enhance performance. To improve outcomes, the assessment must be followed up by analysis of the results and, most importantly, by action.

Medical management clinical guidelines, whether developed internally or purchased and then adjusted to meet specified objectives and local practice standards, are an extension of the organization's overall philosophies and goals. It is essential to the future viability of the plan or provider organization, the welfare of members and corporate partners, and the organization's standing in the community that they be applied appropriately and in a uniform fashion. Periodic benchmarking of clinical guideline application via an inter-rater reliability study is one way for an organization to insure its intentions for utilization management are met.

This paper will examine several elements that health plans or provider groups with utilization management responsibilities should consider when designing and implementing an inter-rater reliability study. The example presented illustrates some aspects of the process. The example, although fairly simple, demonstrates how easily an inter-rater reliability study can be performed. However, inter-rater reliability is a complex concept, and a much more detailed analysis is possible. End users of any inter-rater reliability analysis should be advised of the method and depth of the analysis to avoid confusion or misunderstandings.

Managed Care’s Top Ten Articles of 2016

There’s a lot more going on in health care than mergers (Aetna-Humana, Anthem-Cigna) creating huge players. Hundreds of insurers operate in 50 different states. Self-insured employers, ACA public exchanges, Medicare Advantage, and Medicaid managed care plans crowd an increasingly complex market.

Major health care players are determined to make health information exchanges (HIEs) work. The push toward value-based payment alone almost guarantees that HIEs will be tweaked, poked, prodded, and overhauled until they deliver on their promise. The goal: straight talk from and among tech systems.

They bring a different mindset. They’re willing to work in teams and focus on the sort of evidence-based medicine that can guide health care’s transformation into a system based on value. One question: How well will this new generation of data-driven MDs deal with patients?

The surge of new MS treatments have been for the relapsing-remitting form of the disease. There’s hope for sufferers of a different form of MS. By homing in on CD20-positive B cells, ocrelizumab is able to knock them out and other aberrant B cells circulating in the bloodstream.

A flood of tests have insurers ramping up prior authorization and utilization review. Information overload is a problem. As doctors struggle to keep up, health plans need to get ahead of the development of the technology in order to successfully manage genetic testing appropriately.

Having the data is one thing. Knowing how to use it is another. Applying its computational power to the data, a company called RowdMap puts providers into high-, medium-, and low-value buckets compared with peers in their markets, using specific benchmarks to show why outliers differ from the norm.
Competition among manufacturers, industry consolidation, and capitalization on me-too drugs are cranking up generic and branded drug prices. This increase has compelled PBMs, health plan sponsors, and retail pharmacies to find novel ways to turn a profit, often at the expense of the consumer.
The development of recombinant DNA and other technologies has added a new dimension to care. These medications have revolutionized the treatment of rheumatoid arthritis and many of the other 80 or so autoimmune diseases. But they can be budget busters and have a tricky side effect profile.

Shelley Slade
Vogel, Slade & Goldstein

Hub programs have emerged as a profitable new line of business in the sales and distribution side of the pharmaceutical industry that has got more than its fair share of wheeling and dealing. But they spell trouble if they spark collusion, threaten patients, or waste federal dollars.

More companies are self-insuring—and it’s not just large employers that are striking out on their own. The percentage of employers who fully self-insure increased by 44% in 1999 to 63% in 2015. Self-insurance may give employers more control over benefit packages, and stop-loss protects them against uncapped liability.