Results Summary

PCORI funded the Pilot Projects to explore how to conduct and use patient-centered outcomes research in ways that can better serve patients and the healthcare community. Learn more.


Comparative effectiveness research seeks to determine whether one treatment for a disease works better than another treatment. It compares the effectiveness of different treatments in people with the same disease. Researchers can use large data sets to conduct observational comparative effectiveness research, which is research that doesn’t require recruiting patients to participate in a randomized controlled trial. However, using observational data can produce biased results if the patients receiving one treatment differ from the patients receiving the other treatment in certain ways.

Project Purpose

This study compared several statistical methods to see which ones were best at removing the effects of four kinds of bias that often occur in comparative effectiveness research. It used health insurance claims data and studied several ways of correcting each kind of bias.


The research team used drug treatments for heart attack as their example of an observational research topic. After a patient has a heart attack, doctors often prescribe a type of medicine called a statin to lower cholesterol. Having low cholesterol reduces the chances of a second heart attack, stroke, and death. Statin drugs vary in strength. Using Medicare claims data, the researchers studied the comparative effectiveness of high- and medium-strength statins in preventing additional heart attacks, strokes, and death following a first heart attack. The research team did this using different approaches to minimize bias. Then they compared the results.

The researchers first selected all Medicare claims for hospital stays for a heart attack in 2009. They included patients that were 66 years old or older who had lived for at least 30 days after leaving the hospital. Then they compared the chances of having another heart attack or a stroke and of dying between patients who received high-strength statins and those receiving medium-strength statins. Finally, the researchers looked for the four kinds of bias and tried different statistical methods to reduce their effects.


The researchers examined four types of bias.

Confounding by indication: Doctors often prescribe a high-strength statin because patients are expected to have a poor outcome. Doctors may also prescribe a moderate-strength statin because patients are expected to have a good outcome. In these cases, researchers might be wrong to conclude that moderate-strength statins have better outcomes because the patients who got moderate-strength statins were expected to do better from the start. This kind of bias is called confounding by indication, because patients’ outcomes depend more on their original diagnosis or health status than on the kind of statin they received. The researchers used a statistical method called instrumental variable analysis to reduce this bias. They found that different ways of using this method produced great differences in the estimates of statin effectiveness. Thus, it was not possible to draw clear conclusions about whether one statistical method was most appropriate to address confounding by indication.

Selection bias due to nonpersistence: If a statin does not work well for someone, they are more likely than someone who is happy with the drug to stop using it or to use less than the prescribed dosage. This is called nonpersistence or nonadherence. If patients are more likely to stop taking one type of statin, then it would be wrong to conclude that it doesn’t work as well as another statin. The researchers used a statistical technique called marginal structural modeling to reduce this bias. When researchers did not use this method to minimize bias, they found that moderate-strength statins were more effective than high-strength statins for some outcomes. However, after researchers reduced selection bias due to nonpersistence, they found that high-strength statins were more likely to prevent death. Thus, minimizing selection bias changed their conclusion.

Heterogeneity of treatment effects: Different groups of people often respond differently to the same treatment. For example, women might respond differently than men to the same statin. If researchers group women and men together, then they might miss this difference, which is called heterogeneity of treatment effects. Overall, the researchers found that patients who received high-strength statins were slightly less likely to die than patients who received moderate-strength statins. This finding was similar to the finding of a randomized controlled trial that attempted to answer the same question. The researchers tried two statistical approaches to identify heterogeneity of treatment effects:

  • Conducting the analysis separately for each of several subgroups defined by race/ethnicity, gender, co-occurring disease, and prior statin use
  • Using the overall relative effect multiplied by the specific risk identified for each subgroup

They found that the second approach was better at accounting for heterogeneity of treatment effects than the subgroup analysis approach.

Mortality as a competing risk: When researchers compare the effects of high- and medium-strength statins on the chance of having a second heart attack, dying is called a competing risk, because people in the study who die can’t have a second heart attack. If the chance of dying in the high-strength statin group is the same as in the medium-strength group, there is no bias of this kind. But if the chance of dying differs in the two groups, the results are biased by death as a competing risk. Researchers use special statistical methods called competing risk analyses” to minimize this problem. This study added another type of method called propensity score methods to see if they provided even better control of bias. They found that combining competing risk analyses methods with either propensity score matching or propensity score weighting made the high- and medium-strength statin groups more similar to each other, thereby reducing this kind of bias.


The researchers had to assume certain things about the data that they could not be sure were true. They used sensitivity analysis to test whether these assumptions affected their results, but they cannot be certain that data from other sources on other topics would produce the same results about each kind of bias.

Importance of Findings

Understanding which methods work best for reducing the effects of these four kinds of bias and knowing when to use them is essential if researchers are to draw appropriate conclusions about the comparative effectiveness of treatment. Treatment changes, heterogeneity, and competing risks affect both randomized trials and nonexperimental studies.

Sharing the Results

The researchers have submitted articles about the study to scientific journals.

Project Information

Til Sturmer, PhD, MD
University of North Carolina at Chapel Hill
Methods to Increase Validity of Comparative Effectiveness Research in the Elderly

Key Dates

June 2012
December 2014

Study Registration Information


Has Results
Award Type
Funding Type
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: April 11, 2024