Results Summary
PCORI funded the Pilot Projects to explore how to conduct and use patient-centered outcomes research in ways that can better serve patients and the healthcare community. Learn more.
Background
Comparative effectiveness research seeks to determine whether one treatment for a disease works better than another treatment. It compares the effectiveness of different treatments in people with the same disease. Researchers can use large data sets to conduct observational comparative effectiveness research, which is research that doesn’t require recruiting patients to participate in a randomized controlled trial. However, using observational data can produce biased results if the patients receiving one treatment differ from the patients receiving the other treatment in certain ways.
Project Purpose
This study compared several statistical methods to see which ones were best at removing the effects of four kinds of bias that often occur in comparative effectiveness research. It used health insurance claims data and studied several ways of correcting each kind of bias.
Methods
The research team used drug treatments for heart attack as their example of an observational research topic. After a patient has a heart attack, doctors often prescribe a type of medicine called a statin to lower cholesterol. Having low cholesterol reduces the chances of a second heart attack, stroke, and death. Statin drugs vary in strength. Using Medicare claims data, the researchers studied the comparative effectiveness of high- and medium-strength statins in preventing additional heart attacks, strokes, and death following a first heart attack. The research team did this using different approaches to minimize bias. Then they compared the results.
The researchers first selected all Medicare claims for hospital stays for a heart attack in 2009. They included patients that were 66 years old or older who had lived for at least 30 days after leaving the hospital. Then they compared the chances of having another heart attack or a stroke and of dying between patients who received high-strength statins and those receiving medium-strength statins. Finally, the researchers looked for the four kinds of bias and tried different statistical methods to reduce their effects.
Findings
The researchers examined four types of bias.
Confounding by indication: Doctors often prescribe a high-strength statin because patients are expected to have a poor outcome. Doctors may also prescribe a moderate-strength statin because patients are expected to have a good outcome. In these cases, researchers might be wrong to conclude that moderate-strength statins have better outcomes because the patients who got moderate-strength statins were expected to do better from the start. This kind of bias is called confounding by indication, because patients’ outcomes depend more on their original diagnosis or health status than on the kind of statin they received. The researchers used a statistical method called instrumental variable analysis to reduce this bias. They found that different ways of using this method produced great differences in the estimates of statin effectiveness. Thus, it was not possible to draw clear conclusions about whether one statistical method was most appropriate to address confounding by indication.
Selection bias due to nonpersistence: If a statin does not work well for someone, they are more likely than someone who is happy with the drug to stop using it or to use less than the prescribed dosage. This is called nonpersistence or nonadherence. If patients are more likely to stop taking one type of statin, then it would be wrong to conclude that it doesn’t work as well as another statin. The researchers used a statistical technique called marginal structural modeling to reduce this bias. When researchers did not use this method to minimize bias, they found that moderate-strength statins were more effective than high-strength statins for some outcomes. However, after researchers reduced selection bias due to nonpersistence, they found that high-strength statins were more likely to prevent death. Thus, minimizing selection bias changed their conclusion.
Heterogeneity of treatment effects: Different groups of people often respond differently to the same treatment. For example, women might respond differently than men to the same statin. If researchers group women and men together, then they might miss this difference, which is called heterogeneity of treatment effects. Overall, the researchers found that patients who received high-strength statins were slightly less likely to die than patients who received moderate-strength statins. This finding was similar to the finding of a randomized controlled trial that attempted to answer the same question. The researchers tried two statistical approaches to identify heterogeneity of treatment effects:
- Conducting the analysis separately for each of several subgroups defined by race/ethnicity, gender, co-occurring disease, and prior statin use
- Using the overall relative effect multiplied by the specific risk identified for each subgroup
They found that the second approach was better at accounting for heterogeneity of treatment effects than the subgroup analysis approach.
Mortality as a competing risk: When researchers compare the effects of high- and medium-strength statins on the chance of having a second heart attack, dying is called a competing risk, because people in the study who die can’t have a second heart attack. If the chance of dying in the high-strength statin group is the same as in the medium-strength group, there is no bias of this kind. But if the chance of dying differs in the two groups, the results are biased by death as a competing risk. Researchers use special statistical methods called competing risk analyses” to minimize this problem. This study added another type of method called propensity score methods to see if they provided even better control of bias. They found that combining competing risk analyses methods with either propensity score matching or propensity score weighting made the high- and medium-strength statin groups more similar to each other, thereby reducing this kind of bias.
Limitations
The researchers had to assume certain things about the data that they could not be sure were true. They used sensitivity analysis to test whether these assumptions affected their results, but they cannot be certain that data from other sources on other topics would produce the same results about each kind of bias.
Importance of Findings
Understanding which methods work best for reducing the effects of these four kinds of bias and knowing when to use them is essential if researchers are to draw appropriate conclusions about the comparative effectiveness of treatment. Treatment changes, heterogeneity, and competing risks affect both randomized trials and nonexperimental studies.
Sharing the Results
The researchers have submitted articles about the study to scientific journals.
Professional Abstract
PCORI funded the Pilot Projects to explore how to conduct and use patient-centered outcomes research in ways that can better serve patients and the healthcare community. Learn more.
Background
Comparative effectiveness research seeks to determine if one treatment for a disease works better than another treatment. It compares the effectiveness of different treatments in people living with the same disease. Results from such studies are usually of great relevance but may suffer from biases because of their observational nature and the lack of a randomized intervention. Bias occurs because researchers are not able to completely separate the effect of an intervention (medication) on an outcome (e.g., risk of heart attack or acute myocardial infarction (AMI)) from underlying patient differences.
Project Purpose
The researchers studied four methodological issues that occur with observational data by comparative effectiveness data from studies on treatments after AMI and the risk of subsequent cardiovascular events and mortality. The methodological issues that were examined were: confounding by indication, selection bias due to nonpersistence, heterogeneity of treatment effects, and mortality as a competing risk.
Study Design
Retrospective cohort study.
Participants, Interventions, Settings, and Outcomes
Patients were Medicare beneficiaries 66 years and older who had fee-for-service Medicare coverage, were hospitalized for AMI in the year 2008, and survived at least 30 days after discharge. Interventions consisted of secondary prevention of cardiovascular morbidity and mortality after AMI with guideline-recommended drug therapy focusing on high- or moderate-potency statins using nonexperimental design. The settings were acute care hospitals and community settings nationwide. Outcome measures were a composite of subsequent AMI, stroke, or heart failure requiring hospitalization and all-cause mortality up to two years after hospital discharge (except for the competing risk analyses).
Data Sources
Sources were Medicare insurance claims at least 12 months before and through the end of the study period (December 31, 2009, or death) after the index AMI hospitalization.
Data Analysis
Researchers used instrumental variables (IVs) based on national, regional, and healthcare setting variations in treatment modalities to address confounding by indication. Marginal structural modeling approaches were used to account for informative treatment changes (i.e., drug discontinuation and nonadherence) that occurred during follow-up. Methods to routinely assess heterogeneity of treatment effects on the more relevant numbers needed to treat scale in settings were used where background risk varies by subgroup. Researchers also extended standardization methods to deal with competing risks. They used multivariable techniques developed in the randomized controlled trial setting to assess treatment effectiveness, employing observational data sets from diverse populations.
The inherent limitations of claims data for comparative effectiveness research (CER) are not discussed here because they are at the very core of this methodological project. For all aims, researchers performed extensive sensitivity analyses to assess alternative explanations. The researchers did not collect supplemental data because they sought to assess Medicare claims data as they are typically applied in CER.
Findings
- Confounding by indication: The strength of the instruments and IVs varied considerably. The ability of the IVs and their different specifications to balance baseline covariates also varied markedly. The estimates of treatment effectiveness across the various IVs were also markedly different with substantial differences in the standard errors.
- Selection bias due to nonpersistence: Among “statin users,” 62.9% were moderate-potency users. Sixty-seven percent of moderate-potency users and 70% of high-potency users discontinued by the end of follow-up (median time to discontinuation approximately 110 days), and 9% of moderate-potency and 13% of high-potency users switched initial potencies. The intention-to-treat analysis indicated no benefit of high‑potency statins. The as-treated analysis suggested low-potency statins were more beneficial for some outcomes, although these findings were likely subject to selection bias. The time-varying, weighted approach addressing selection bias indicated that high-potency statin use was associated with decreased all-cause mortality (hazard ratio: 0.90 [0.84, 0.98]) but no other cardiovascular outcomes.
- Mortality as a competing risk: High- versus moderate‑potency statin use after myocardial infarction (MI) was associated with a slight decrease in the risk of all-cause mortality within one year (-6.8/1000; 95% CI: -15.4, 1.7) with a corresponding number needed to treat of 147 and a 6% relative reduction in mortality within one year (adjusted hazard ratio = 0.94, 95% CI: 0.86, 1.02). This estimate is similar to that obtained in randomized trials among patients aged 65 or older.
- Heterogeneity of treatment effects: Alternative methods of assessing heterogeneous treatment effects (HTE) on the absolute scale identified different possible sources of HTE. Estimates of the expected risk difference based on the estimated baseline risks and assuming that the relative effect was homogeneous were more stable than subgroup‑specific analyses. Using this method, the researchers identified meaningful (50% difference from the overall population) HTE in the effect of high-potency statins on liver/kidney injury by race/ethnicity, diabetes at baseline, and history of statin use prior to the index MI.
- Mortality as a competing risk: Researchers established the utility of propensity score methods in competing risks analyses with multiple event types. Matching and weighting methods were shown to perform well numerically, yielding reduced bias relative to the standard analyses. Substantial gains were evident in analyses of statin effectiveness at the population level, where death and cardiovascular events are the competing risks. These gains were particularly strong in older populations, where healthy individuals are much more likely to receive treatment.
Limitations
All analytic strategies implemented rely on untestable assumptions. However, sensitivity analyses helped assess the effect of violations of these assumptions.
Conclusions
Results indicate that advanced study design and analytic methods of Medicare claims data can clarify the benefit of secondary prevention after AMI. Nonexperimental studies will continue to be a large part of the evidence base for clinical decision making, and developing and disseminating methods to increase the validity of nonexperimental studies is important.