Real World Evidence

Observational Studies Fill a Void Left by RCTs

The health system is entering uncharted territory with this approach, and new rules might be needed to ensure the studies are credible.

Thomas Reinke
Contributing Editor

New life is being breathed into real-world evidence. The idea of incorporating information collected from circumstances closer to clinical reality has always had intuitive appeal. But real-world evidence has been widely criticized as sounding good but, in actuality, may be unreliable or result in biased results compared to randomized trials.

New data sources, more rigorous study designs, more powerful analytics, and smarter, more appropriate use of studies are lending real-world data some new credibility.

Payers, pharma getting along

Jennifer Graff

Drug companies and payers are designing real-world evidence studies together that dig deeper into how medications are used and for whom they are effective, says Jennifer Graff of the National Pharmaceutical Council.

For decades, the pharmaceutical industry has used the randomized clinical trial to generate reliable evidence of the safety and efficacy of its drugs. Now the observational study has become a way for companies to build a body of evidence that supports commercialization of their products, identifies opportunities for new agents, and streamlines late-stage clinical trials. Of course observational studies are nothing new. They are the workhorses of nutrition and public health research. But there is momentum building to use them to study the safety and efficacy of medications in new and novel ways. Drug companies are using observational studies to establish the value of their medicines in negotiating prices, rebates, and formulary placement with payers. With crowding in many drug classes, payers have the leverage to demand that a drug performs better than others. Health systems are using observational studies to figure out if the outcomes from the phase 3 trials of newer medicines are generalizable to broader populations or, perhaps, specific population subgroups. Rigorous observational studies and pragmatic clinical trials (see box below), which are conducted in a variety of settings and compare experimental treatments to existing treatments, are now widely accepted comparative effectiveness studies.

What is a pragmatic clinical trial?

Pragmatic clinical trials are designed to evaluate the effectiveness of interventions in real-life routine practice conditions, whereas randomized controlled trials (RCTs) are typically designed to test the efficacy of intervention under optimal situations for demonstrating a treatment effect. Pragmatic trials produce results that can be generalized and applied in routine practice settings.

In the past, drug companies and payers often had a contentious relationship. Now, they are designing real-world evidence studies together that dig into how medications are used and for whom they are effective, says Jennifer Graff, a vice president with the National Pharmaceutical Council. She says there are a number of publicly announced partnership arrangements to analyze data, and more that have not been publicized.

Real-world lung study

A prime example is the study that GlaxoSmithKline (GSK) conducted recently with Great Britain’s National Health Service (NHS). The Salford Lung Study was a community-based phase 3 trial of GSK’s Breo Ellipta, a combination of fluticasone furoate, an inhaled corticosteroid, and vilanterol, a long-acting beta2-adrenergic agonist, that was approved as a treatment for chronic obstructive pulmonary disease (COPD). The pragmatic clinical trial, which had few exclusions, included 2,800 patients from 80 general practices plus 130 pharmacies. Extensive data about their total care was drawn from their electronic health records. The study compared patients receiving the GSK agent with patients receiving usual care with several other COPD medications. The goal of the yearlong study was to get as complete a picture as possible of the way patients use their medication, the care they receive on a regular basis for COPD, and additional care they receive for other conditions.

The primary outcome was the rate of moderate or severe exacerbations among patients who had had an exacerbation within one year before the trial. Secondary outcomes were the rates of primary care contact (contact with a general practitioner, nurse, or other health care professional) and secondary care contact (inpatient admission, outpatient visit with a specialist, or visit to the emergency department).

There was a statistically significant reduction in the rate of moderate or severe exacerbations in patients treated with Breo Ellipta compared with patients receiving usual care.

Typically, randomized clinical trials are designed to test medications in very controlled circumstances with carefully selected patients. By contrast, the Salford Lung Study had more of an all-comers approach with lots of data for researchers to analyze. Drug manufacturers are eyeing high quality, pragmatic trials like this because they can generate data about a drug working (or not working) in specific populations. Perhaps a medication is safer in specific age groups or is most effective when it’s paired with another drug.

Besides, the Salford Lung Study benefited the NHS as much as it did GSK by providing insight into patient behaviors, the utilization of services, and other aspects of their care. The NHS praised the study as a pioneering model that sets a precedent for future studies. The goal is to capture data about the performance of medications in everyday clinical situations rather than conduct a controlled experiment, which, in essence, is what a randomized clinical trial is.

Head-to-head comparison

One real-world design of particular interest to providers is the retrospective head-to-head comparisons of different medications. Randomized trials have been used for these comparisons, but they are expensive to conduct—and funding is an issue. The NIH has other priorities, and drug companies don’t want to run the risk of their product not coming out on top. Yet head-to-head comparisons would help providers make better decisions about which drugs to prescribe and, ultimately, might improve patient outcomes. Head-to-head studies can help providers generalize from a very narrow phase 3 trial to a broader population or to identify differences among subpopulations.

Researchers at the Mayo Clinic recently published two head-to-head observational studies of the anticoagulants dabigatran (Pradaxa), rivaroxaban (Xarelto), and apixaban (Eliquis). In phase 3 trials, all three have been shown to be safe and as effective as warfarin, the mainstay of anticoagulants. Warfarin requires frequent lab tests and dosing adjustments, so alternatives are desirable, but some important data on this trio of newcomers—particularly on the risk of major bleeding—has been lacking.

Peter Noseworthy, MD

In comparison to multisite randomized controlled trials, [head-to-head] studies are easier to organize and a good deal less expensive, says Peter Noseworthy, MD, of the Mayo Clinic.

One of the Mayo studies, published online by Chest, was a head-to-head comparison of the three rivals to warfarin. Several indirect comparisons have been published but no direct comparisons, says Peter Noseworthy, MD, an author of both studies.

The study in Chest included data on more than 57,000 patients pulled from a large insurance claims database. Noseworthy and his colleagues found no statistically significant difference among the three agents as far as preventing strokes and other efficacy outcomes. They also looked at inpatient admissions for gastrointestinal bleeding, intracranial bleeding, and bleeding from other sites as a marker of safety. Apixaban was associated with less major bleeding than dabigatran and rivaroxaban, and rivaroxaban was associated with an increased risk of major bleeding and intracranial bleeding compared with dabigatran.

The second Mayo study, published in the Journal of the American Heart Association, compared each of the three relatively new anticoagulants directly to warfarin. The researchers had data on more than 76,000 patients. The results showed apixaban was associated with lower risks of both stroke and major bleeding than warfarin, dabigatran was associated with a similar risk of stroke but lower risk of major bleeding, and rivaroxaban was associated with similar risks of both stroke and major bleeding.

Noseworthy says the head-to-head studies help providers understand the safety and efficacy of the three drugs—and perhaps help them with prescription decisions. Studying the three medications (four if you count warfarin) together eliminates the work and uncertainty of interpreting multiple trials with different populations and methodologies.

In comparison to multisite randomized controlled trials, these types of studies are easier to organize and a good deal less expensive, says Noseworthy. The complexity is in the analytical methods. “We don’t have the opportunity to prospectively control the study so we have to use sophisticated methods to reduce or eliminate the biases,” he says.

An example of bias, he says, is prescriber preferences. Prescribers have preferences for which drug they prescribed for different kinds of patients. Retrospective studies must account for that or run the risk of attributing results to the drugs that are really the result of patient characteristics. The bias can be reduced through propensity score matching which was used in both studies, says Noseworthy. Once bias and confounding variables are reduced, Noseworthy says the amount of work that goes into analysis of data and determination of results is relatively straightforward.

However, Noseworthy cautions that a single study cannot provide definitive information for generalizing the use of a medication across a broad population or within a specific subpopulation. “We don’t want to make sweeping generalizations from a single study. Instead several studies are needed to fill the gaps that exist in phase 3 trials.”

A checklist

Jonathan Morris, MD, a vice president with QuintilesIMS, says that additional patient data sources like registries, better study designs, and more powerful analytic techniques require broader new methods for generating information for decision making. “Different study designs are appropriate for answering different questions and the proper strategy is to match the study design to the level of question or problem that needs to be answered.”

Because observational studies are still emerging as a study design for pharmaceuticals, new rules of the road may be needed. GRACE Principles (Good ReseArch for Comparative Effectiveness), a nonprofit consortium to improve the quality of observational studies, has come out with a tool called the GRACE Checklist. It was spearheaded by QuintilesIMS and the National Pharmaceutical Council. The checklist includes 11 questions about data and methods such as, Were the outcomes studied valid in other populations? Validation activities have documented the usefulness of all 11 questions in this checklist (see box below).

What makes for a quality observational study?

QuintilesIMS and the National Pharmaceutical Council are leading the Good ReseArch for Comparative Effective (GRACE) initiative. The GRACE checklist poses 11 questions to ask about an observational trial that shed some light on whether it is a high-quality study.

  1. Were treatment and/or important details of treatment exposure adequately recorded for the study purpose in the data source(s)?
  2. Were the primary outcomes adequately recorded for the study purpose (e.g., available in sufficient detail through data source(s))?
  3. Was the primary clinical outcome(s) measured objectively rather than subject to clinical judgment (e.g., opinion about whether the patient’s condition has improved)?
  4. Were primary outcomes validated, adjudicated, or otherwise known to be valid in a similar population?
  5. Was the primary outcome(s) measured or identified in an equivalent manner between the treatment/intervention group and the comparison group(s)?
  6. Were important covariates that may be known confounders or effect modifiers available and recorded?
  7. Was the study (or analysis) population restricted to new initiators of treatment or those starting a new course of treatment?
  8. If one or more comparison groups were used, were they concurrent comparators? If not, did the authors justify the use of historical comparison group(s)?
  9. Were important covariates, confounding and effect modifying variables taken into account in the design and/or analysis?
  10. Is the classification of exposed and unexposed person-time free of “immortal time bias”?
  11. Were any meaningful analyses conducted to test key assumptions on which primary results are based?

Morris says that does not negate the need for new study designs to fill in for the limitations of random controlled trials. “In 1933, President Roosevelt moved our monetary system off the gold standard and that boosted the economic recovery. Our health care system would benefit greatly if it moved beyond random controlled trials and included other research designs.”

A blueprint for high-volume, high-quality lung cancer screening that is detecting cancer earlier—and helping to save lives

Clinical Brief

Multiple Sclerosis: New Perspectives on the Patient Journey–2019 Update
Summary of an Actuarial Analysis and Report