Results Summary
What was the research about?
Comparative effectiveness research compares two or more treatments to see which one works better for which patients. In some studies, researchers assign patients by chance to several treatments or to have or not have a treatment. But approaches that assign patients by chance are not always suitable. For example, assigning patients to a new treatment may not be good medical care.
For this reason, researchers sometimes do studies using data collected when patients and their doctors choose the treatments. Data from such studies are observational data. When using observational data for research, it can be hard to know if the effect of a treatment is because of the treatment or other factors, such as patients’ age, gender, or health history. In these cases, researchers use statistical methods to understand the effect of the treatment. Depending on the study’s focus and design, some methods work better than others.
In this study, the research team developed guidance for researchers to help them choose methods for their study.
What were the results?
The research team created an online guide. The guide explains the differences between various methods for doing research and gives options for analyzing observational data. It also includes links to other articles and websites. The guide could help researchers choose the method that would be the right fit for their study.
What did the research team do?
First, the research team searched for published articles that describe methods for using observational data to analyze the effects of treatments. Next, they analyzed data to compare the methods to see how they did for finding out treatment effects. Finally, the research team used these results to develop the online guide for other researchers.
What were the limits of the study?
The research team looked only at certain types of methods and studies. For example, the team looked only at articles for studies that compared two treatments. Searching for other types of studies may have led to different ideas for the guide.
Future studies could expand the guide to include other methods. The team could also update the article list in the guide as new research becomes available.
How can people use the results?
Researchers using observational data to compare treatments can use the guide to decide on the design and analysis for their research. Results from studies that use the guide can help patients and doctors compare treatments.
Professional Abstract
Objective
To develop guidance for researchers for selecting and applying appropriate statistical methods to analyze observational data in comparative effectiveness research (CER)
Study Design
Design Elements | Description |
---|---|
Design | Systematic review, simulation studies |
Data Sources and Data Sets | PubMed, EMASE, PsycINFO, and Current Index to Statistics |
Analytic Approach | Literature search, development of decision guide |
Outcomes | DECODE CER decision guide |
Clinical researchers analyzing observational data face challenges in trying to determine which statistical analysis options are appropriate for supporting causal inferences, given their specific questions and data. Options include propensity score matching; weighting and subclassification; doubly robust methods; instrumental variables; and methods for handling data with treatment regimens that could change over time, or time-varying treatments. Studies indicate that the method used may affect the findings. For results that are consistent with the goals of a study, it is important to align an appropriate method to the study question and data.
To guide researchers, the research team conducted a systematic review of the literature, identifying important considerations in selecting and applying statistical methods for causal inference with observational data. In addition, the team conducted simulation studies to investigate the statistical properties of propensity-score-based approaches. Results from the systematic review and simulations guided the development of a decision guide to aid researchers in choosing appropriate methods for CER.
In the systematic review, the research team identified studies of causal inference methods specific to the use of observational data in CER. The review focused on studies with a single binary treatment and a single binary outcome. The team searched the PubMed, EMASE, PsycINFO, and Current Index to Statistics databases for simulation studies or theoretical findings that assessed control of bias and precision in analyzing observational data. Simulation studies compared five different propensity-score-based methods.
Researchers with expertise in statistics, social epidemiology, health policy, outcomes effectiveness research, pharmacy, and physical therapy provided input on literature search strategies, analytic issues, and the decision guide.
Results
The systematic review identified 10,342 possible articles for review, of which 168 met the criteria for inclusion in the final analysis. Of the studies, 62% used propensity-score-based methods; the rest used other causal inference methods, such as instrumental variables, alone or in combination. The simulation studies revealed that the propensity-score-based methods performed better than methods using logistic models.
The research team formatted the decision guide they created as a series of interactive slides titled Decision Tool for Causal Inference and Observational Data Analysis Methods in Comparative Effectiveness Research (DECODE CER). This decision guide directs researchers through options to select methods that are suitable for their research-specific objectives and data sets. It contains information on, and links to, relevant information sources, literature references, and educational websites.
Limitations
The literature review did not include purely empirical studies, review articles, or studies focused on treatment dose levels or multivariate outcomes. Including other studies may have led to different conclusions.
DECODE CER focuses primarily on two types of causal inference methods and does not accommodate research designs beyond binary outcome and binary treatment. In addition, the guide does not provide details about examining subpopulations. These constraints limit applicability.
Conclusions and Relevance
This study highlighted the statistical properties of analytic methods that are useful for CER that uses observational data.
The research team developed a publicly available interactive decision guide to inform researchers in selecting and applying optimal methods for a given data set and specific CER questions.
Future Research Needs
Future research could expand DECODE CER to encompass a wider variety of methods. Researchers could update the guide as new publications appear. Future expansions could incorporate methods for handling data with time-varying treatments or analysis of subpopulations.
Final Research Report
View this project's final research report.
Journal Citations
Related Journal Citations
Stories and Videos
Videos
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- Reviewers generally found the study to be very strong and the report to be clearly written but could not see the connection between the systematic review and the development of the decision tool. The researchers explained that instead of summarizing the articles described in the systematic review, they used the information from the review to develop the final structure of the decision tool.
- Reviewers asked how the research accomplished one of the original objectives of the systematic review, which was determining which methodological technique works best when. The researchers revised the report to explain that they focused instead on the need to align the research question, methods, and interpretation in observational research.