Skip to main content
Patient-Centered Outcomes Research Institute
Patient-Centered Outcomes Research Institute
  • Blog
  • Newsroom
  • Find It Fast
  • Help Center
  • Subscribe
  • Careers
  • Contact Us

PCORI

Patient-Centered Outcomes Research Institute

Search form

  • About Us
    Close mega-menu

    About Us

    • Our Programs
    • Governance
    • Financials and Reports
    • Procurement Opportunities
    • Our Staff
    • Our Vision & Mission
    • Contact Us

    Fact Sheets: Learn More About PCORI

    Download fact sheets about out work, the research we fund, and our programs and initiatives.

    Find It Fast

    Browse through an alphabetical list of frequently accessed and searched terms for information and resources.

    Subscribe to PCORI Email Alerts

    Sign up for weekly emails to stay current on the latest results of our funded projects, and more.

  • Research & Results
    Close mega-menu

    Research & Results

    • Explore Our Portfolio
    • Research Fundamentals
    • Research Results Highlights
    • Putting Evidence to Work
    • Peer Review
    • Evidence Synthesis
    • About Our Research

    Evidence Updates from PCORI-Funded Studies

    These updates capture highlights of findings from systematic reviews and our funded research studies.

    Journal Articles About Our Funded Research

    Browse through a collection of journal publications that provides insights into PCORI-funded work.

    Explore Our Portfolio of Funded Projects

    Find out about projects based on the health conditions they focus on, the state they are in, and if they have results.

  • Topics
    Close mega-menu

    Topics

    • Addressing Disparities
    • Arthritis
    • Asthma
    • Cancer
    • Cardiovascular Disease
    • Children's Health
    • Community Health Workers
    • COVID-19
    • Dementia and Cognitive Impairment
    • Diabetes
    • Kidney Disease
    • Medicaid
    • Men's Health
    • Mental and Behavioral Health
    • Minority Mental Health
    • Multiple Chronic Conditions
    • Multiple Sclerosis
    • Obesity
    • Older Adults' Health
    • Pain Care and Opioids
    • Rare Diseases
    • Rural Health
    • Shared Decision Making
    • Telehealth
    • Transitional Care
    • Veterans Health
    • Women's Health

    Featured Topic: Women's Health

    Learn more about the projects we support on conditions that specifically or more often affect women.

  • Engagement
    Close mega-menu

    Engagement

    • The Value of Engagement
    • Engagement in Health Research Literature Explorer
    • Influencing the Culture of Research
    • Engagement Awards
    • Engagement Resources
    • Engage with Us

    Engagement Tools and Resources for Research

    This searchable peer-to-peer repository includes resources that can inform future work in patient-centered outcomes research.

    Explore Engagement in Health Literature

    This tool enables searching for published articles about engagement in health research.

    Research Fundamentals: A New On-Demand Training

    It enables those new to health research or patient-centered research to learn more about the research process.

  • Funding Opportunities
    Close mega-menu

    Funding Opportunities

    • What & Who We Fund
    • What You Need to Know to Apply
    • Applicant Training
    • Merit Review
    • Awardee Resources
    • Help Center

    PCORI Funding Opportunities

    View and learn about the newly opened funding announcements and the upcoming PFAs in 2021.

    Tips for Submitting a Responsive LOI

    Find out what PCORI looks for in a letter of intent (LOI) along with other helpful tips.

    PCORI Awardee Resources

    These resources can help awardees in complying with the terms and conditions of their contract.

  • Meetings & Events
    Close mega-menu

    Meetings & Events

    • Upcoming
    • Past Events

    PCORI Webinar: February 2, 2pm ET

    Hear from PCORI leaders about ways to get involved in PCOR, improvements to our funding opportunities, and more. Register

    Confronting COVID-19: A Webinar Series

    Learn more about the series and access recordings and summary reports of all six sessions.

    2020 PCORI Annual Meeting

    Watch recordings of all sessions, and view titles and descriptions of the posters presented at the virtual meeting.

You are here

  • Research & Results
  • Explore Our Portfolio
  • Developing an Interactive Online Guid...

This project has results

Developing an Interactive Online Guide to Support the Use of Causal Inference Methods in Comparative Effectiveness Research

Sign Up for Updates to This Study  

Results Summary and Professional Abstract

Results Summary

Results Summary

Download Summary

What was the research about?

Comparative effectiveness research compares two or more treatments to see which one works better for which patients. One type of research study is a randomized controlled trial, or an RCT. In an RCT, the research team assigns patients to a treatment by chance.

Other types of studies use information from health records and registries. Registries store data about patients with a specific health problem. They often include information on how each patient responds to a treatment. Because researchers don’t assign treatments by chance in such studies, differences in how patients respond to a treatment may be from the treatment or something else, such as a patient’s age or the severity of their illness. In studies using registries and health records, researchers apply statistical approaches, called causal inference methods, to estimate how treatments work. At the same time, they look at other things that could affect results, like a patient’s age.

Researchers can choose among many different causal inference methods. But they may have a hard time knowing which methods to use or how to use complex methods correctly. In this study, the research team made an interactive online guide for researchers. The guide, called CERBOT, helps researchers design studies and select these methods.

What were the results?

The research team created an online guide called CERBOT, which has five sections. Researchers enter information about the study they would like to do on the CERBOT website. Then CERBOT creates a report with information on how to design studies using information from health records and registries. It suggests what statistical methods to use and how to use them.

What did the research team do?

To design CERBOT, the research team asked for input from researchers, statisticians, and patients. The group suggested what the guide should do, how it should work, and how to make it easy to use. The research team also looked at other research studies designed using methods like those considered by CERBOT. With this information, the team created CERBOT and then tested how well it worked.

What were the limits of the study?

Researchers can use CERBOT for some study designs, but not for others.

Future research could improve CERBOT. For example, researchers could add more features to it. The creators of CERBOT could improve the guide so that it works for additional study designs.

How can people use the results?

Researchers can use CERBOT to design comparative effectiveness studies based on information from health registries and health records. Studies designed using these methods could provide information to doctors and patients about treatments.

Professional Abstract

Professional Abstract

Objective

To develop a web-based interactive guide for (1) formulating a well-defined comparative effectiveness research (CER) question and study design using observational data and (2) selecting an appropriate causal inference analytical method

Study Design

Design Elements Description
Design Qualitative study and literature review
Data Sources and Data Sets Input from an 8-member advisory committee; literature review
Analytic Approach Qualitative analysis and targeted literature synthesis
Outcomes Interactive online guide to support using the target trial framework and causal inference methods in CER

When randomized controlled trials (RCTs) are not feasible for a specific research question, researchers can use observational data to emulate a hypothetical randomized trial, known as the target trial. Despite advances in causal inference methods for observational studies that emulate RCTs, these complex methods have been largely inaccessible to applied researchers or clinicians who work with observational data. Common challenges include how to implement causal inference study design principles and how to select suitable analytic methods based on specific questions and data. Options for analytical methods include g-methods, doubly robust methods, instrumental variables, propensity scoring, and standard conditioning methods.

To address this gap, the research team developed a web-based guide called Comparative Effectiveness Research Based on Observational Data to Emulate a Target Trial (CERBOT). The guide aids researchers using observational data for CER by explicitly specifying and emulating a target trial. CERBOT also helps researchers create a complete study design and select appropriate causal inference methods for analysis.

The research team convened an eight-member advisory committee of researchers, statisticians, patient representatives, and dissemination experts to provide input on CERBOT’s aim, scope, and functionality as well as approaches for making it accessible to researchers. The research team also conducted a targeted literature review on comparative effectiveness studies that applied the target trial framework. The team then qualitatively analyzed the committee’s input and results from the literature review to refine and finalize the conceptual framework for CERBOT.

Results

The completed conceptual framework defined two key steps for causal inference in CER using observational data. First, formulate a causal question by designing the ideal hypothetical RCT, the target trial. Then, emulate the design and analysis of the target trial using observational data.

The CERBOT guide takes users through five modules where they input information about their intended research. They provide eligibility criteria, outcomes, follow-up period, treatment strategies, and confounding variables. Upon completion of the modules, users receive a report with recommendations for an appropriate study design and analytical methods for causal inference.

Limitations

CERBOT requires additional formal evaluation and testing with researchers from various fields using different observational data to ensure content clarity, usefulness, and ease of use. Moreover, CERBOT may not be applicable under certain complex study design scenarios (e.g., multiple or complex treatment strategies, eligibility criteria met at multiple time points).

Conclusions and Relevance

CERBOT may provide researchers and clinicians with a user-friendly application to guide them in designing and analyzing CER using observational data to emulate RCTs. It may expand researchers’ abilities to perform causal inference using observational data and enable non-statisticians to use causal inference methods to explore the effects of treatments.

Future Research Needs

Researchers need to conduct further evaluation and testing and apply CERBOT in different and complex designs to assess its performance and adapt it to different settings.

Final Research Report

View this project's final research report.

Journal Articles

More on this Project  

PCORI Stories

Improving Methods for Studying Chronic Disease
A narrative on how researchers are creating an easy-to-use statistical toolkit to help researchers study treatments that vary over time.

Peer-Review Summary

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:

  • The reviewers questioned the researchers’ use of a target trial—a hypothetical trial structure used to test causal inference—in testing their CERBOT instrument. The reviewers noted that consensus is lacking about the need for a target trial to establish causal inference in observational data. Moreover, this framework does not fit all causal questions, like those involving safety or less easily manipulated exposures. The researchers added to their justification for using a target trial. They also confirmed that the CERBOT instrument’s design enables it to measure comparative effectiveness and test safety.
  • Reviewers identified several areas where the description of the CERBOT tool itself was difficult to understand. The researchers made extensive revisions based on these comments. They also added examples and a video tutorial to the CERBOT website to help users implement the tool.
  • Reviewers expressed concern that without a well-defined causal question, the tool would not be useful in developing the components of a target trial. The researchers agreed and revised the report to say that a well-defined question would be one for which an investigator could specify a hypothetical research trial.
  • Reviewers asked for clarification of the makeup of the study’s advisory committee which the researchers claimed to include stakeholders and patient representatives. The researchers noted that the committee included clinicians working with patients but acknowledged that it lacked members that represented a patient perspective.

Conflict of Interest Disclosures

View the COI disclosure form.

Project Details

Principal Investigator
Yi Zhang, PhD, MS
Project Status
Completed; PCORI Public and Professional Abstracts, and Final Research Report Posted
Project Title
Development of a Causal Inference Toolkit for Patient-Centered Outcomes Research
Board Approval Date
September 2013
Project End Date
December 2018
Organization
Medical Technology and Practice Patterns Institute
Year Awarded
2013
State
Maryland
Year Completed
2018
Project Type
Research Project
Health Conditions  
Blood Disorders
Anemia
Cancer
Cardiovascular Diseases
Kidney Diseases
Chronic Kidney Disease
Multiple/Comorbid Chronic Conditions
Nutritional and Metabolic Disorders
Diabetes
Intervention Strategies
Drug Interventions
Funding Announcement
Improving Methods for Conducting Patient-Centered Outcomes Research
Project Budget
$1,194,104
DOI - Digital Object Identifier
10.25302/1.2020.13036031
Study Registration Information
HSRP20143592
Page Last Updated: 
January 31, 2020

About Us

  • Our Programs
  • Governance
  • Financials and Reports
  • Procurement Opportunities
  • Our Staff
  • Our Vision & Mission
  • Contact Us

Research & Results

  • Explore Our Portfolio
  • Research Fundamentals
  • Research Results Highlights
  • Putting Evidence to Work
  • Peer Review
  • Evidence Synthesis
  • About Our Research

Engagement

  • The Value of Engagement
  • Engagement in Health Research Literature Explorer
  • Influencing the Culture of Research
  • Engagement Awards
  • Engagement Resources
  • Engage with Us

Funding Opportunities

  • What & Who We Fund
  • What You Need to Know to Apply
  • Applicant Training
  • Merit Review
  • Awardee Resources
  • Help Center

Meetings & Events

February 2
PCORI 2021 and Beyond: Opportunities for Funding and Involvement in Patient-Centered Research
February 9
Board of Governors Meeting: February 9, 2021
February 11
Advisory Panel on Patient Engagement Winter 2021 Meeting

PCORI

Footer contact address

Patient-Centered Outcomes
Research Institute

1828 L Street, NW, Suite 900
Washington, DC 20036
Phone: (202) 827-7700 | Fax: (202) 355-9558
[email protected]

Subscribe to Newsletter

Twitter Facebook LinkedIn Vimeo

© 2011-2021 Patient-Centered Outcomes Research Institute. All Rights Reserved.

Privacy Policy | Terms of Use | Trademark Usage Guidelines | Credits | Help Center