Skip to main content
Patient-Centered Outcomes Research Institute
Patient-Centered Outcomes Research Institute
  • Blog
  • Newsroom
  • Help Center
  • Subscribe
  • Careers
  • Contact

PCORI

Patient-Centered Outcomes Research Institute

Search form

  • About Us
    Close mega-menu

    About Us

    • Our Programs
    • Governance
    • Financials and Reports
    • Procurement Opportunities
    • Our Staff
    • Contact Us

    Subscribe to PCORI Email Alerts

    Sign up for weekly emails to stay current on the latest results of our funded projects, and more.

    Learn More About PCORI

    Download fact sheets about our work, the research we fund, and our programs and initiatives.

    Join Us! Work at PCORI

    View our current openings, benefits and apply online. Featured Opening: Program Officer

  • Research & Results
    Close mega-menu

    Research & Results

    • Explore Our Portfolio
    • Research Results Highlights
    • PCORI in the Literature
    • Putting Evidence to Work
    • Peer Review
    • Evidence Synthesis
    • About Our Research

    Highlights of PCORI-Funded Research Results

    Read about results that are being published in leading medical journals and presented at major scientific meetings.

    CE/CME Activities: Treatment Options After a Stroke

    Find out how to apply research findings to help patients with atrial fibrillation make informed decisions about treatments following an ischemic stroke.

    Explore Our Portfolio of Funded Projects

    Find out about projects based on the health conditions they focus on, the state they are in, and if they have results.

  • Topics
    Close mega-menu

    Topics

    • Addressing Disparities
    • Cancer
    • Cardiovascular Disease
    • Community Health Workers
    • Dementia and Cognitive Impairment
    • Diabetes
    • Kidney Disease
    • Mental and Behavioral Health
    • Multiple Chronic Conditions
    • Multiple Sclerosis
    • Obesity
    • Older Adults' Health
    • Pain Care and Opioids
    • Rare Diseases
    • Shared Decision Making
    • Telehealth
    • Transitional Care

    Featured Topic: Multiple Chronic Conditions

    A PCORI-funded study shows that a family-centered communication program in pediatric departments directly improves hospital safety.

  • Engagement
    Close mega-menu

    Engagement

    • The Value of Engagement
    • Engagement in Health Research Literature Explorer
    • Influencing the Culture of Research
    • Engagement Awards
    • Engagement Resources
    • Engage with Us

    Older Adults Advise Researchers

    An Engagement Award brings together researchers and older adults living at home and in skilled nursing facilities to improve older adults' care.

    Explore Engagement in Health Literature

    PCORI has developed a new tool that enables searching for published articles about engagement in health research.

    Using the PCORI Engagement Rubric

    It provides general guidance and suggestions for engagement compliance and engagement strategies in the research process.

  • Funding Opportunities
    Close mega-menu

    Funding Opportunities

    • What & Who We Fund
    • What You Need to Know to Apply
    • Applicant Training
    • Merit Review Process
    • Awardee Resources
    • Help Center

    PCORI Funding Announcements

    Learn about open and upcoming funding announcements and how to apply.

    Tips for Submitting a Responsive LOI

    Find out what PCORI looks for in a letter of intent (LOI) along with other helpful tips.

    Do You Have Questions?

    PCORI's Help Center includes an FAQ section for answers to applicant, awardee and reviewer questions.

  • Meetings & Events
    Close mega-menu

    Meetings & Events

    • Upcoming
    • Past Events

    Watch Our Facebook Live Discussion

    We discussed how telehealth might help improve the outcomes for patients living with serious mental illness.

    Hear from PCORI Staff

    Invite a PCORI staffer to talk about our work and see a schedule of our staffers' speaking engagements.

    From Evidence to Impact: Putting What Works into Action

    Watch archived recordings of our 2018 Annual Meeting's keynotes, plenaries and breakout sessions.

You are here

  • Research & Results
  • Explore Our Portfolio
  • Guidance for Researchers on Optimal M...

Guidance for Researchers on Optimal Methods for Conducting Comparative Effectiveness Research with Observational Data

This project has results

Abstracts

Public Abstract

Download this abstract: English (pdf)

What was the research about?

Comparative effectiveness research compares two or more treatments to see which one works better for which patients. In some studies, researchers assign patients by chance to several treatments or to have or not have a treatment. But approaches that assign patients by chance are not always suitable. For example, assigning patients to a new treatment may not be good medical care.

For this reason, researchers sometimes do studies using data collected when patients and their doctors choose the treatments. Data from such studies are observational data. When using observational data for research, it can be hard to know if the effect of a treatment is because of the treatment or other factors, such as patients’ age, gender, or health history. In these cases, researchers use statistical methods to understand the effect of the treatment. Depending on the study’s focus and design, some methods work better than others.

In this study, the research team developed guidance for researchers to help them choose methods for their study.

What were the results?

The research team created an online guide. The guide explains the differences between various methods for doing research and gives options for analyzing observational data. It also includes links to other articles and websites. The guide could help researchers choose the method that would be the right fit for their study.

What did the research team do?

First, the research team searched for published articles that describe methods for using observational data to analyze the effects of treatments. Next, they analyzed data to compare the methods to see how they did for finding out treatment effects. Finally, the research team used these results to develop the online guide for other researchers.

What were the limits of the study?

The research team looked only at certain types of methods and studies. For example, the team looked only at articles for studies that compared two treatments. Searching for other types of studies may have led to different ideas for the guide.

Future studies could expand the guide to include other methods. The team could also update the article list in the guide as new research becomes available.

How can people use the results?

Researchers using observational data to compare treatments can use the guide to decide on the design and analysis for their research. Results from studies that use the guide can help patients and doctors compare treatments.

Professional Abstract

Objective

To develop guidance for researchers for selecting and applying appropriate statistical methods to analyze observational data in comparative effectiveness research (CER)

Study Design

Design Elements Description
Design Systematic review, simulation studies
Data Sources and Data Sets PubMed, EMASE, PsycINFO, and Current Index to Statistics
Analytic Approach Literature search, development of decision guide
Outcomes DECODE CER decision guide

Clinical researchers analyzing observational data face challenges in trying to determine which statistical analysis options are appropriate for supporting causal inferences, given their specific questions and data. Options include propensity score matching; weighting and subclassification; doubly robust methods; instrumental variables; and methods for handling data with treatment regimens that could change over time, or time-varying treatments. Studies indicate that the method used may affect the findings. For results that are consistent with the goals of a study, it is important to align an appropriate method to the study question and data.

To guide researchers, the research team conducted a systematic review of the literature, identifying important considerations in selecting and applying statistical methods for causal inference with observational data. In addition, the team conducted simulation studies to investigate the statistical properties of propensity-score-based approaches. Results from the systematic review and simulations guided the development of a decision guide to aid researchers in choosing appropriate methods for CER.

In the systematic review, the research team identified studies of causal inference methods specific to the use of observational data in CER. The review focused on studies with a single binary treatment and a single binary outcome. The team searched the PubMed, EMASE, PsycINFO, and Current Index to Statistics databases for simulation studies or theoretical findings that assessed control of bias and precision in analyzing observational data. Simulation studies compared five different propensity-score-based methods.

Researchers with expertise in statistics, social epidemiology, health policy, outcomes effectiveness research, pharmacy, and physical therapy provided input on literature search strategies, analytic issues, and the decision guide.

Results

The systematic review identified 10,342 possible articles for review, of which 168 met the criteria for inclusion in the final analysis. Of the studies, 62% used propensity-score-based methods; the rest used other causal inference methods, such as instrumental variables, alone or in combination. The simulation studies revealed that the propensity-score-based methods performed better than methods using logistic models.

The research team formatted the decision guide they created as a series of interactive slides titled Decision Tool for Causal Inference and Observational Data Analysis Methods in Comparative Effectiveness Research (DECODE CER). This decision guide directs researchers through options to select methods that are suitable for their research-specific objectives and data sets. It contains information on, and links to, relevant information sources, literature references, and educational websites.

Limitations

The literature review did not include purely empirical studies, review articles, or studies focused on treatment dose levels or multivariate outcomes. Including other studies may have led to different conclusions.

DECODE CER focuses primarily on two types of causal inference methods and does not accommodate research designs beyond binary outcome and binary treatment. In addition, the guide does not provide details about examining subpopulations. These constraints limit applicability.

Conclusions and Relevance

This study highlighted the statistical properties of analytic methods that are useful for CER that uses observational data.

The research team developed a publicly available interactive decision guide to inform researchers in selecting and applying optimal methods for a given data set and specific CER questions.

Future Research Needs

Future research could expand DECODE CER to encompass a wider variety of methods. Researchers could update the guide as new publications appear. Future expansions could incorporate methods for handling data with time-varying treatments or analysis of subpopulations.

This project's final research report is expected to be available by June 2019.

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:

  • Reviewers generally found the study to be very strong and the report to be clearly written but could not see the connection between the systematic review and the development of the decision tool. The researchers explained that instead of summarizing the articles described in the systematic review, they used the information from the review to develop the final structure of the decision tool.
  • Reviewers asked how the research accomplished one of the original objectives of the systematic review, which was determining which methodological technique works best when. The researchers revised the report to explain that they focused instead on the need to align the research question, methods, and interpretation in observational research.

Conflict of Interest Disclosures

View the COI disclosure form.

More on this Project  

Videos

Including Stakeholders in Research
Although skeptical at first, this statistician  included some unusual stakeholders in his methods research.

Project Details

Principal Investigator
Douglas Landsittel, PhD
Project Status
Completed; PCORI Public and Professional Abstracts Posted
Project Title
Modeling Strategies for Observational Comparative Effectiveness Research - What Works Best When?
Board Approval Date
December 2013
Project End Date
October 2018
Organization
University of Pittsburgh at Pittsburgh
Year Awarded
2013
State
Pennsylvania
Year Completed
2018
Project Type
Research Project
Funding Announcement
Improving Methods for Conducting Patient-Centered Outcomes Research
Project Budget
$1,057,002
Study Registration Information
HSRP20143604
Page Last Updated: 
October 3, 2018
Sign Up for Updates to This Study 
  • Top of Page

About Us

  • Our Programs
  • Governance
  • Financials and Reports
  • Procurement Opportunities
  • Our Staff
  • Contact Us

Research & Results

  • Explore Our Portfolio
  • Research Results Highlights
  • PCORI in the Literature
  • Putting Evidence to Work
  • Peer Review
  • Evidence Synthesis
  • About Our Research

Engagement

  • The Value of Engagement
  • Engagement in Health Research Literature Explorer
  • Influencing the Culture of Research
  • Engagement Awards
  • Engagement Resources
  • Engage with Us

Funding Opportunities

  • What & Who We Fund
  • What You Need to Know to Apply
  • Applicant Training
  • Merit Review Process
  • Awardee Resources
  • Help Center

Meetings & Events

February 26
Board of Governors Meeting: February 26, 2019
March 19
Board of Governors Meeting: March 19, 2019
April 16
Board of Governors Meeting: April 16, 2019

PCORI

Footer contact address

Patient-Centered Outcomes
Research Institute

1828 L Street, NW, Suite 900
Washington, DC 20036
Phone: (202) 827-7700 | Fax: (202) 355-9558
info@pcori.org

Subscribe to Newsletter

Twitter Facebook LinkedIn Vimeo

© 2011-2019 Patient-Centered Outcomes Research Institute. All Rights Reserved.

Privacy Policy | Terms of Use | Trademark Usage Guidelines | Credits | Help Center