• Download Summary

What was the research about?

Comparative effectiveness research compares two or more treatments to see which one works better for which patients. In some studies, researchers assign patients by chance to several treatments or to have or not have a treatment. But approaches that assign patients by chance are not always suitable. For example, assigning patients to a new treatment may not be good medical care.

For this reason, researchers sometimes do studies using data collected when patients and their doctors choose the treatments. Data from such studies are observational data. When using observational data for research, it can be hard to know if the effect of a treatment is because of the treatment or other factors, such as patients’ age, gender, or health history. In these cases, researchers use statistical methods to understand the effect of the treatment. Depending on the study’s focus and design, some methods work better than others.

In this study, the research team developed guidance for researchers to help them choose methods for their study.

What were the results?

The research team created an online guide. The guide explains the differences between various methods for doing research and gives options for analyzing observational data. It also includes links to other articles and websites. The guide could help researchers choose the method that would be the right fit for their study.

What did the research team do?

First, the research team searched for published articles that describe methods for using observational data to analyze the effects of treatments. Next, they analyzed data to compare the methods to see how they did for finding out treatment effects. Finally, the research team used these results to develop the online guide for other researchers.

What were the limits of the study?

The research team looked only at certain types of methods and studies. For example, the team looked only at articles for studies that compared two treatments. Searching for other types of studies may have led to different ideas for the guide.

Future studies could expand the guide to include other methods. The team could also update the article list in the guide as new research becomes available.

How can people use the results?

Researchers using observational data to compare treatments can use the guide to decide on the design and analysis for their research. Results from studies that use the guide can help patients and doctors compare treatments.

Objective

To develop guidance for researchers for selecting and applying appropriate statistical methods to analyze observational data in comparative effectiveness research (CER)

Study Design

Clinical researchers analyzing observational data face challenges in trying to determine which statistical analysis options are appropriate for supporting causal inferences, given their specific questions and data. Options include propensity score matching; weighting and subclassification; doubly robust methods; instrumental variables; and methods for handling data with treatment regimens that could change over time, or time-varying treatments. Studies indicate that the method used may affect the findings. For results that are consistent with the goals of a study, it is important to align an appropriate method to the study question and data.

To guide researchers, the research team conducted a systematic review of the literature, identifying important considerations in selecting and applying statistical methods for causal inference with observational data. In addition, the team conducted simulation studies to investigate the statistical properties of propensity-score-based approaches. Results from the systematic review and simulations guided the development of a decision guide to aid researchers in choosing appropriate methods for CER.

In the systematic review, the research team identified studies of causal inference methods specific to the use of observational data in CER. The review focused on studies with a single binary treatment and a single binary outcome. The team searched the PubMed, EMASE, PsycINFO, and Current Index to Statistics databases for simulation studies or theoretical findings that assessed control of bias and precision in analyzing observational data. Simulation studies compared five different propensity-score-based methods.

Researchers with expertise in statistics, social epidemiology, health policy, outcomes effectiveness research, pharmacy, and physical therapy provided input on literature search strategies, analytic issues, and the decision guide.

Results

The systematic review identified 10,342 possible articles for review, of which 168 met the criteria for inclusion in the final analysis. Of the studies, 62% used propensity-score-based methods; the rest used other causal inference methods, such as instrumental variables, alone or in combination. The simulation studies revealed that the propensity-score-based methods performed better than methods using logistic models.

The research team formatted the decision guide they created as a series of interactive slides titled Decision Tool for Causal Inference and Observational Data Analysis Methods in Comparative Effectiveness Research (DECODE CER). This decision guide directs researchers through options to select methods that are suitable for their research-specific objectives and data sets. It contains information on, and links to, relevant information sources, literature references, and educational websites.

Limitations

The literature review did not include purely empirical studies, review articles, or studies focused on treatment dose levels or multivariate outcomes. Including other studies may have led to different conclusions.

DECODE CER focuses primarily on two types of causal inference methods and does not accommodate research designs beyond binary outcome and binary treatment. In addition, the guide does not provide details about examining subpopulations. These constraints limit applicability.

Conclusions and Relevance

This study highlighted the statistical properties of analytic methods that are useful for CER that uses observational data.

The research team developed a publicly available interactive decision guide to inform researchers in selecting and applying optimal methods for a given data set and specific CER questions.

Future Research Needs

Future research could expand DECODE CER to encompass a wider variety of methods. Researchers could update the guide as new publications appear. Future expansions could incorporate methods for handling data with time-varying treatments or analysis of subpopulations.