Objective
To develop guidance for researchers for selecting and applying appropriate statistical methods to analyze observational data in comparative effectiveness research (CER)
Study Design
| Design Elements |
Description |
| Design |
Systematic review, simulation studies |
| Data Sources and Data Sets |
PubMed, EMASE, PsycINFO, and Current Index to Statistics |
| Analytic Approach |
Literature search, development of decision guide |
| Outcomes |
DECODE CER decision guide |
Clinical researchers analyzing observational data face challenges in trying to determine which statistical analysis options are appropriate for supporting causal inferences, given their specific questions and data. Options include propensity score matching; weighting and subclassification; doubly robust methods; instrumental variables; and methods for handling data with treatment regimens that could change over time, or time-varying treatments. Studies indicate that the method used may affect the findings. For results that are consistent with the goals of a study, it is important to align an appropriate method to the study question and data.
To guide researchers, the research team conducted a systematic review of the literature, identifying important considerations in selecting and applying statistical methods for causal inference with observational data. In addition, the team conducted simulation studies to investigate the statistical properties of propensity-score-based approaches. Results from the systematic review and simulations guided the development of a decision guide to aid researchers in choosing appropriate methods for CER.
In the systematic review, the research team identified studies of causal inference methods specific to the use of observational data in CER. The review focused on studies with a single binary treatment and a single binary outcome. The team searched the PubMed, EMASE, PsycINFO, and Current Index to Statistics databases for simulation studies or theoretical findings that assessed control of bias and precision in analyzing observational data. Simulation studies compared five different propensity-score-based methods.
Researchers with expertise in statistics, social epidemiology, health policy, outcomes effectiveness research, pharmacy, and physical therapy provided input on literature search strategies, analytic issues, and the decision guide.
Results
The systematic review identified 10,342 possible articles for review, of which 168 met the criteria for inclusion in the final analysis. Of the studies, 62% used propensity-score-based methods; the rest used other causal inference methods, such as instrumental variables, alone or in combination. The simulation studies revealed that the propensity-score-based methods performed better than methods using logistic models.
The research team formatted the decision guide they created as a series of interactive slides titled Decision Tool for Causal Inference and Observational Data Analysis Methods in Comparative Effectiveness Research (DECODE CER). This decision guide directs researchers through options to select methods that are suitable for their research-specific objectives and data sets. It contains information on, and links to, relevant information sources, literature references, and educational websites.
Limitations
The literature review did not include purely empirical studies, review articles, or studies focused on treatment dose levels or multivariate outcomes. Including other studies may have led to different conclusions.
DECODE CER focuses primarily on two types of causal inference methods and does not accommodate research designs beyond binary outcome and binary treatment. In addition, the guide does not provide details about examining subpopulations. These constraints limit applicability.
Conclusions and Relevance
This study highlighted the statistical properties of analytic methods that are useful for CER that uses observational data.
The research team developed a publicly available interactive decision guide to inform researchers in selecting and applying optimal methods for a given data set and specific CER questions.
Future Research Needs
Future research could expand DECODE CER to encompass a wider variety of methods. Researchers could update the guide as new publications appear. Future expansions could incorporate methods for handling data with time-varying treatments or analysis of subpopulations.
