Results Summary

What was the research about?

Comparative effectiveness research compares two or more treatments to see which one works better for which patients. One type of research study is a randomized controlled trial, or an RCT. In an RCT, the research team assigns patients to a treatment by chance.

Other types of studies use information from health records and registries. Registries store data about patients with a specific health problem. They often include information on how each patient responds to a treatment. Because researchers don’t assign treatments by chance in such studies, differences in how patients respond to a treatment may be from the treatment or something else, such as a patient’s age or the severity of their illness. In studies using registries and health records, researchers apply statistical approaches, called causal inference methods, to estimate how treatments work. At the same time, they look at other things that could affect results, like a patient’s age.

Researchers can choose among many different causal inference methods. But they may have a hard time knowing which methods to use or how to use complex methods correctly. In this study, the research team made an interactive online guide for researchers. The guide, called CERBOT, helps researchers design studies and select these methods.

What were the results?

The research team created an online guide called CERBOT, which has five sections. Researchers enter information about the study they would like to do on the CERBOT website. Then CERBOT creates a report with information on how to design studies using information from health records and registries. It suggests what statistical methods to use and how to use them.

What did the research team do?

To design CERBOT, the research team asked for input from researchers, statisticians, and patients. The group suggested what the guide should do, how it should work, and how to make it easy to use. The research team also looked at other research studies designed using methods like those considered by CERBOT. With this information, the team created CERBOT and then tested how well it worked.

What were the limits of the study?

Researchers can use CERBOT for some study designs, but not for others.

Future research could improve CERBOT. For example, researchers could add more features to it. The creators of CERBOT could improve the guide so that it works for additional study designs.

How can people use the results?

Researchers can use CERBOT to design comparative effectiveness studies based on information from health registries and health records. Studies designed using these methods could provide information to doctors and patients about treatments.

Final Research Report

View this project's final research report.

Stories and Videos

Peer-Review Summary

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:

  • The reviewers questioned the researchers’ use of a target trial—a hypothetical trial structure used to test causal inference—in testing their CERBOT instrument. The reviewers noted that consensus is lacking about the need for a target trial to establish causal inference in observational data. Moreover, this framework does not fit all causal questions, like those involving safety or less easily manipulated exposures. The researchers added to their justification for using a target trial. They also confirmed that the CERBOT instrument’s design enables it to measure comparative effectiveness and test safety.
  • Reviewers identified several areas where the description of the CERBOT tool itself was difficult to understand. The researchers made extensive revisions based on these comments. They also added examples and a video tutorial to the CERBOT website to help users implement the tool.
  • Reviewers expressed concern that without a well-defined causal question, the tool would not be useful in developing the components of a target trial. The researchers agreed and revised the report to say that a well-defined question would be one for which an investigator could specify a hypothetical research trial.
  • Reviewers asked for clarification of the makeup of the study’s advisory committee which the researchers claimed to include stakeholders and patient representatives. The researchers noted that the committee included clinicians working with patients but acknowledged that it lacked members that represented a patient perspective.

Conflict of Interest Disclosures

Project Information

Yi Zhang, PhD, MS
Medical Technology and Practice Patterns Institute
$1,170,037 *
Development of a Causal Inference Toolkit for Patient-Centered Outcomes Research

Key Dates

September 2013
December 2018

Study Registration Information

Final Research Report

View this project's final research report.

Journal Articles


Has Results
Award Type
Health Conditions Health Conditions These are the broad terms we use to categorize our funded research studies; specific diseases or conditions are included within the appropriate larger category. Note: not all of our funded projects focus on a single disease or condition; some touch on multiple diseases or conditions, research methods, or broader health system interventions. Such projects won’t be listed by a primary disease/condition and so won’t appear if you use this filter tool to find them. View Glossary
Intervention Strategy Intervention Strategies PCORI funds comparative clinical effectiveness research (CER) studies that compare two or more options or approaches to health care, or that compare different ways of delivering or receiving care. View Glossary
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: January 20, 2023