Results Summary
What was the research about?
Comparative effectiveness research compares two or more treatments to see which one works better for which patients. One type of research study is a randomized controlled trial, or an RCT. In an RCT, the research team assigns patients to a treatment by chance.
Other types of studies use information from health records and registries. Registries store data about patients with a specific health problem. They often include information on how each patient responds to a treatment. Because researchers don’t assign treatments by chance in such studies, differences in how patients respond to a treatment may be from the treatment or something else, such as a patient’s age or the severity of their illness. In studies using registries and health records, researchers apply statistical approaches, called causal inference methods, to estimate how treatments work. At the same time, they look at other things that could affect results, like a patient’s age.
Researchers can choose among many different causal inference methods. But they may have a hard time knowing which methods to use or how to use complex methods correctly. In this study, the research team made an interactive online guide for researchers. The guide, called CERBOT, helps researchers design studies and select these methods.
What were the results?
The research team created an online guide called CERBOT, which has five sections. Researchers enter information about the study they would like to do on the CERBOT website. Then CERBOT creates a report with information on how to design studies using information from health records and registries. It suggests what statistical methods to use and how to use them.
What did the research team do?
To design CERBOT, the research team asked for input from researchers, statisticians, and patients. The group suggested what the guide should do, how it should work, and how to make it easy to use. The research team also looked at other research studies designed using methods like those considered by CERBOT. With this information, the team created CERBOT and then tested how well it worked.
What were the limits of the study?
Researchers can use CERBOT for some study designs, but not for others.
Future research could improve CERBOT. For example, researchers could add more features to it. The creators of CERBOT could improve the guide so that it works for additional study designs.
How can people use the results?
Researchers can use CERBOT to design comparative effectiveness studies based on information from health registries and health records. Studies designed using these methods could provide information to doctors and patients about treatments.
Professional Abstract
Objective
To develop a web-based interactive guide for (1) formulating a well-defined comparative effectiveness research (CER) question and study design using observational data and (2) selecting an appropriate causal inference analytical method
Study Design
| Design Elements | Description |
|---|---|
| Design | Qualitative study and literature review |
| Data Sources and Data Sets | Input from an 8-member advisory committee; literature review |
| Analytic Approach | Qualitative analysis and targeted literature synthesis |
| Interactive online guide to support using the target trial framework and causal inference methods in CER |
When randomized controlled trials (RCTs) are not feasible for a specific research question, researchers can use observational data to emulate a hypothetical randomized trial, known as the target trial. Despite advances in causal inference methods for observational studies that emulate RCTs, these complex methods have been largely inaccessible to applied researchers or clinicians who work with observational data. Common challenges include how to implement causal inference study design principles and how to select suitable analytic methods based on specific questions and data. Options for analytical methods include g-methods, doubly robust methods, instrumental variables, propensity scoring, and standard conditioning methods.
To address this gap, the research team developed a web-based guide called Comparative Effectiveness Research Based on Observational Data to Emulate a Target Trial (CERBOT). The guide aids researchers using observational data for CER by explicitly specifying and emulating a target trial. CERBOT also helps researchers create a complete study design and select appropriate causal inference methods for analysis.
The research team convened an eight-member advisory committee of researchers, statisticians, patient representatives, and dissemination experts to provide input on CERBOT’s aim, scope, and functionality as well as approaches for making it accessible to researchers. The research team also conducted a targeted literature review on comparative effectiveness studies that applied the target trial framework. The team then qualitatively analyzed the committee’s input and results from the literature review to refine and finalize the conceptual framework for CERBOT.
Results
The completed conceptual framework defined two key steps for causal inference in CER using observational data. First, formulate a causal question by designing the ideal hypothetical RCT, the target trial. Then, emulate the design and analysis of the target trial using observational data.
The CERBOT guide takes users through five modules where they input information about their intended research. They provide eligibility criteria, outcomes, follow-up period, treatment strategies, and confounding variables. Upon completion of the modules, users receive a report with recommendations for an appropriate study design and analytical methods for causal inference.
Limitations
CERBOT requires additional formal evaluation and testing with researchers from various fields using different observational data to ensure content clarity, usefulness, and ease of use. Moreover, CERBOT may not be applicable under certain complex study design scenarios (e.g., multiple or complex treatment strategies, eligibility criteria met at multiple time points).
Conclusions and Relevance
CERBOT may provide researchers and clinicians with a user-friendly application to guide them in designing and analyzing CER using observational data to emulate RCTs. It may expand researchers’ abilities to perform causal inference using observational data and enable non-statisticians to use causal inference methods to explore the effects of treatments.
Future Research Needs
Researchers need to conduct further evaluation and testing and apply CERBOT in different and complex designs to assess its performance and adapt it to different settings.
Final Research Report
View this project's final research report.
Stories and Videos
PCORI Stories
Improving Methods for Studying Chronic Disease
Peer-Review Summary
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers questioned the researchers’ use of a target trial—a hypothetical trial structure used to test causal inference—in testing their CERBOT instrument. The reviewers noted that consensus is lacking about the need for a target trial to establish causal inference in observational data. Moreover, this framework does not fit all causal questions, like those involving safety or less easily manipulated exposures. The researchers added to their justification for using a target trial. They also confirmed that the CERBOT instrument’s design enables it to measure comparative effectiveness and test safety.
- Reviewers identified several areas where the description of the CERBOT tool itself was difficult to understand. The researchers made extensive revisions based on these comments. They also added examples and a video tutorial to the CERBOT website to help users implement the tool.
- Reviewers expressed concern that without a well-defined causal question, the tool would not be useful in developing the components of a target trial. The researchers agreed and revised the report to say that a well-defined question would be one for which an investigator could specify a hypothetical research trial.
- Reviewers asked for clarification of the makeup of the study’s advisory committee which the researchers claimed to include stakeholders and patient representatives. The researchers noted that the committee included clinicians working with patients but acknowledged that it lacked members that represented a patient perspective.